acceptodds
Under review as a conference paper at ICLR 2027

Plan Before You Denoise: Semantic Availability in Image Generation

Abstract

A text-to-image model can render a visual answer far more reliably than it can execute the instruction that determines it. On controlled Boolean-program prompts (SYNBOOL), FLUX renders the correct colour on only 27.1% of unresolved instructions at 15 operations (collapsing from 60.8% at 3 ops to 24.8–27.1% for ops), but reaches 80.7% when the answer is supplied. Across program depths, sampling budgets, and two model families, unresolved generation collapses to chance levels on multi-step programs, dominated by superficial prompt-position heuristics. We formalize this execution gap through three complementary computational constraints: (1) Content: grounded in usable information (Xu et al., 2020), an exact decoder-relative velocity-risk law shows that when semantic readouts cannot resolve the instruction, the population-optimal restricted velocity collapses to uninformative drift, forfeiting conditional guidance despite statistically complete input. (2) Adaptive compute: a stylized random-oracle benchmark isolates serial depth from parallel width, establishing that with fewer than adaptive rounds, achieving nontrivial advantage requires total queries. (3) Timing: a finite-Euler controllability bound proves that late disclosure monotonically contracts the remaining Gramian, escalating the theoretical minimum steering energy, while empirical evaluations show that standard linear guidance incurs severe control inefficiency when delayed. Upfront planning resolves this bottleneck by paying the serial span in autoregressive reasoning tokens and caching the visual decision from the very first update. These constraints provide insight into complementary failure modes for one-shot planning (single-pass degradation under spatial load) and iterative painting (compounding routing errors across sequential hops). Combining them into a constraint-guided hybrid PLAN+PAINT strategy pays serial span in an upstream planner and spatial work across incremental rendering passes, boosting exact placement on dense relational grids to 0.623–0.773 (vs. 0.401–0.525 for Plan and 0.159–0.529 for Paint), reaching 0.745 on GenEval spatial prompts (vs. 0.395 unresolved), and achieving 0.770 on T2I-ReasonBench entity prompts (vs. 0.345 unresolved, matching an answer-supplied oracle of 0.775).

Primary area:
generative models
Keywords:
Text-to-Image Generation, Diffusion Models, Flow Matching, Compositional Generation, Instruction Following, Semantic Conditioning, - Chain-of-Thought Planning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oim5OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that text-to-image models fail on prompts that require computation because the answer never reaches the renderer in a usable form. On SynBool, a Boolean program in the prompt selects a shape’s colour. There, FLUX.2-Klein drops from 60.8% at 3 operations to chance (about 25%) at 7 or more, but reaches 80.7% when the answer is given. More denoising steps do not help, and Qwen-Image behaves the same way. They formalize this via three theorems in a toy flow-matching model. The paper then proposes Plan+Paint, where an LLM resolves the instruction first and the renderer adds objects one edit at a time. They report gains on synthetic grids, GenEval, and T2I-ReasonBench.

Critical strengths

S1. A clean diagnosis of instruction following versus drawing. SynBool keeps the visual target fixed and changes only how the answer is given. This separates following the instruction from drawing the image. The controls are convincing. The model draws an injected wrong colour 86.8% of the time, and a shuffled reasoning trace works as well as an ordered one (78.7% vs. 78.1%). So the renderer follows the final decision, not the reasoning. Step sweeps and a second model rule out sampler- or model-specific effects. SynBool could be a useful diagnostic tool for others.

S2. A simple error decomposition. End-to-end accuracy is roughly planner accuracy times rendering accuracy on correct plans, and the renderer almost never fixes a wrong plan (r−≈0r_- \approx 0). This gives a clear rule for where to spend test-time compute in such pipelines.

Critical weaknesses

W1. The main remedy is already standard, and the theory makes no testable prediction. Using a language model to resolve or rewrite a prompt before rendering is already common. DALL-E 3 rewrites prompts with an LLM (Betker et al., 2023), and the paper itself cites planner-based and reasoning-first systems such as RPG, LMD, T2I-R1 and GoT. The contribution therefore rests on whether the theory explains something new. However, each theorem is used to explain results after they are observed rather than to predict them. The theory does not explain why accuracy collapses between 3 and 7 operations. The Fourier analysis in Appendix B.1 could predict which SynBool programs a simple readout can solve, but the paper never compares such predictions with per-task accuracy. Without a prediction that the data could have contradicted, the theory restates the empirical finding rather than explaining it.

W2. Theorem 1 is not connected to a real model. Theorem 1 says a model that cannot represent the answer cannot use it, which is nearly true by construction. Its relevance therefore depends on knowing what the model can represent, but the paper never identifies or measures this for FLUX or Qwen-Image. The 60.8% accuracy at 3 operations also shows that the model does compute part of the answer.

W3. The assumptions of Theorem 2 do not hold for FLUX. Theorem 2 assumes the model has fewer sequential steps than the program depth. FLUX runs dozens of layers 28 times, which is far more than the depth of these programs. The bound also assumes each step is a hidden lookup table, whereas SynBool writes every step in the prompt.

W4. The “16× more energy” result follows directly from the guidance gain. Energy grows with the square of the guidance gain. Raising the gain from 1 to 4 therefore gives 42=164^2 = 16 times more energy at any disclosure time. This number says nothing about timing and should not be presented as support for Theorem 3.

W5. The timing experiments do not test Theorem 3. Theorem 3 concerns an optimal control and implies that late correction remains possible at finite cost. The experiments use standard guidance instead. This guidance fails at J=21J=21 even with 4.65× the energy of early guidance. That points to the model committing to colour and layout early, which the theory does not model.

W6. A simpler explanation is not ruled out. The same Qwen3 model stays at chance when it answers without chain-of-thought, and the text-encoder probes collapse with depth. A single LLM forward pass failing on deep Boolean programs is a known result (Feng et al., 2023a; Li et al., 2024). This explains the main finding without any diffusion-specific theory.

W7. The evidence for an extra gap inside the renderer is weak. The only evidence that the renderer adds its own gap is at shallow depth, where the probe reaches 0.927 but rendering reaches 0.646. This probe is a four-layer Transformer that can compute the last gates itself. The linear and MLP probes described in Appendix C.4 would show what is directly readable, but their results are not reported.

W8. No experiment separates “cannot” from “was never trained to”. The paper claims the failure is structural, but current models were never trained on such prompts. Fine-tuning on unresolved SynBool prompts would test this. If shallow programs improve while deep ones stay at chance, the claim would be supported. Without this experiment, both readings fit the data equally well. Running this experiment, and the reframing it may require, goes beyond what a rebuttal can fix.

W9. The compute budgets of the compared methods are not matched. Plan+Paint renders up to five times per image while Plan renders once, and no compute cost is reported. Part of the gain may come from the extra render passes. Comparisons at equal budget are needed, for example Plan with best-of-N sampling and a verifier.

W10. Important baselines are missing. A layout-grounded renderer (LMD, GLIGEN) might solve the crowding problem in a single pass, which would directly test the paper’s explanation for why Plan fails. Existing planner and editing systems (LMD, RPG, SLD, GenArtist) are also not compared, although Plan+Paint is close to GenArtist and SLD.

W11. The T2I-ReasonBench results contradict the paper’s own explanation. These prompts contain a single entity and no crowding, so incremental painting should add little. Yet Plan+Paint beats Plan by 22 points, and Plan is 23 points below the oracle. This suggests the gain comes from differences in prompts or render passes rather than from the span/work trade-off.

W12. The natural-prompt benchmarks do not test serial reasoning. GenEval position prompts contain a single spatial relation. The T2I-ReasonBench entity prompts mostly require recognising one entity from its description. Neither involves the serial dependencies the paper is built around. The gains of Plan on these benchmarks likely come from clearer, more explicit prompts rather than from resolving serial depth, so they do not support the theory.

Minor issues: The step sweep is reported as 24.8–27.1% in Section 3.3, but 24.3–27.1% in Appendix A.3. Figure 5c shows 28.5% while the text says 28.7%. Several items are missing or undefined, including the 16-cell table behind “15 of 16 cells”, the per-cell SynBool results, the “early7” condition, and the flow-control task setup. The symbol σ\sigma means both noise level and σ-algebra, and KJ(a)K_J(a) is never defined. The title says “Plan Before You Denoise” while the method is called “Plan Before You Draw”. The planner’s token budget is not given, so truncation could explain the 31.2% planner accuracy at 31 operations. Please refine the writing. It was very hard to understand and appreciate your contributions due to the poor presentation.

Questions for authors

Q1. What does the probe predict, and what is its chance level? Figure 2 marks 0.5, but rendering is four-way with chance 0.25. If the probe is four-way, 0.481 at deep strata is well above chance, which changes Finding 4. Please also report the linear and MLP probes, the layer used, and the number of tasks per depth level.

Q2. Why does Plan+Paint beat Plan by 22 points on single-entity prompts, and what do its edit steps do there? Please report planner accuracy, the compiled prompts, the planner and editor models for each benchmark, and how FLUX.2-Klein performs the edits.

Q3. What is the total cost of each method (render calls, tokens, time)? How does Plan with best-of-N and a verifier compare at equal cost?

Q4. Which readout class describes FLUX, and can it be measured? How does Theorem 2’s assumption of fewer steps than program depth hold for FLUX?

Q5. Unresolved SynGrid-2×2 accuracy rises from 0.081 to 0.360 between 3 and 4 objects. Is this a scoring or label-balance artefact? Please also add per-object planner and oracle-steps accuracy to Table 1.

2 comments

Thank you for the detailed response and the revised manuscript. Several of my concerns are now resolved. The probe clarifications answer Q1, and the revised timing section, together with the new patching test, addresses W4 and W5. The fine-tuning study, the cost table with 6-step PLAN+PAINT, the per-task Fourier analysis and the explanation of the 2×2 jump (Q5) also make the paper stronger, and the writing is much clearer.

My main concerns, however, are about the core claims, and the new results make them sharper rather than weaker.

The first is where the failure actually happens. The response agrees that deep rules fail because of the known limit of a single Transformer pass (response to W6). The new linear probe points the same way. It reaches 0.784 per bit at shallow depth, which is roughly 0.61 for both bits and close to the 0.646 rendering accuracy. In other words, the renderer seems to read about as much as a linear probe can read from the text encoder. The diffusion-side effects listed in the response (prompt interference, crowding, late disclosure) are real, but they concern drawing the answer, not working it out. The execution gap therefore appears to come from the text encoder rather than from denoising.

The second concern is about the theory. The new evidence does not support the span-versus-work view behind Theorem 2. In Study 2 (Table 5), span is fixed at 2, yet rendering already falls to 29.4% at 7 operations, below the matched chain (39.6%), while the probe stays at 0.92. Rendering failure thus tracks the number of XOR operations, not serial span. Section 3.3 also leaves this result out and reports only the probe. Since the response now explains the timing effect through early commitment rather than the Gramian bound, the theory adds little beyond the observation that the DiT reads text features linearly.

The third concern is the T2I-ReasonBench comparison. The revised Section 5.2 explains the 22-point gain (W11), but the explanation suggests the comparison is not fair. Plan puts the entity, clues and rubric details into one long caption, while PLAN+PAINT draws a short subject caption and then refines it in a second pass. The gain may therefore come from the shorter caption and the extra pass rather than from planning.

The numbers point the same way. The planner is correct on 83.5% of prompts, and wrong plans are rarely rendered correctly (Section 3.5). To reach 0.770, PLAN+PAINT would need to render about 92% of correct plans correctly. That is well above the single-pass oracle that names the entity directly (0.775). W12 also remains open, since neither natural benchmark requires serial reasoning.

Finally, the response agrees that prompt rewriting is standard and that LMD, GLIGEN and RPG share the same division of work as PLAN+PAINT. What remains new is placing objects across several passes instead of within one. This choice is never compared with a single-pass layout-grounded method. Training-free regional prompting already runs on FLUX-style DiTs, so a fair comparison on the same backbone is feasible.

The following experiments would help clarify the contribution:

  1. On T2I-ReasonBench, Plan using the same short caption as the first PLAN+PAINT pass, and the explicit-entity oracle with the same second refinement pass.
  2. A single-pass regional prompting baseline on FLUX.2-Klein for SynGrid-3×3.
  3. Training-set accuracy and the training setup for the fine-tuning runs. Seven operations is far below the depth of the text encoder, so a fixed-depth limit should not apply at this scale. The deep failure may reflect what LoRA learned rather than what the model can represent.
  4. The degree-1 Fourier weight of each span condition in Study 1, to separate span from readout degree.

I appreciate the effort that went into this revision. Since my remaining concerns are about the core claims rather than details, I keep my score for now, but I am happy to continue the discussion. I am inclined to increase my score if these very central issues can be addressed properly. Thanks again.

Thank you for running all the experiments I suggested and for the candid response. The regional prompting baseline and the fine-tuning details are useful additions, and the paper is clearer than the original submission.

However, the new results confirm my central concerns rather than resolve them. The response itself now steps back from the paper's main claims.

  1. The failure is in the text encoder, not in denoising. The response agrees that the bottleneck is text conditioning, and the joint linear probe (0.622) nearly matches rendering (0.646). This is the known limit of a single Transformer pass, so the central finding is not new.
  2. The theory no longer explains the results. The response now presents the theorems as intuition. Fourier weight explains most of the span effect (the L=1 vs. L=3 gap drops from 21.8 to 8.2 points). Yet the theory's own quantity improves out-of-distribution prediction by only 0.019 nats, against 0.094 for the empirical probe.
  3. The method gains are small once baselines are fair. On T2I-ReasonBench, a short-caption Plan already reaches 0.745, so the second pass adds only 2.5 points, and a matched oracle reaches 0.880 to 0.900. On SynGrid, PLAN+PAINT beats regional prompting by about 7 points at matched latency, and only on dense synthetic grids. Planning before rendering is already standard, as the response agrees.
  4. The "cannot" claim is not supported. Joint fine-tuning fits 82.6% of 7-operation training circuits but only 27.7% of test circuits. This shows poor generalization, not a capacity limit.

The current manuscript still makes the claims the response concedes, namely that the theorems explain the gap, that PLAN+PAINT matches the explicit-entity oracle, and that a single-pass capacity limit exists. Aligning the paper with these findings would require a substantial reframing.

What remains is a useful synthetic diagnostic and an incremental planning-plus-editing pipeline, which I do not think meets the high bar of ICLR. I therefore keep my score. Thank you again for the constructive discussion.

Reviewer_Z5JKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the gap between executing an instruction and rendering its visual answer. SynBool holds the visual task simple while varying the computation needed to identify the target color. The paper develops analyses of restricted semantic readouts, adaptive oracle queries, and disclosure timing, and evaluates a Plan+Paint pipeline that resolves dependencies before incremental rendering.

Critical strengths

The controlled interventions provide a useful diagnosis of the gap between computing an answer and rendering it. At 15 operations, FLUX achieves 27.1% accuracy on unresolved instructions, 80.7% when supplied the answer, and 96.7% with a minimal visual prompt. Wrong-answer and shuffled-derivation controls strengthen this interpretation (Sections 3.1–3.3). This is the principal reason I favor acceptance.

The semantic-readout and disclosure-timing analyses offer a useful framework for interpreting this gap, and Plan+Paint improves over both planning alone and iterative painting on relational grids, with supporting transfer experiments (Sections 4–5). Together, these results make the diagnosis relevant to system design.

Critical weaknesses

The theoretical settings are informative mechanism models, but their connection to the evaluated pretrained networks remains indirect. The random-oracle result does not establish a round lower bound for explicit Boolean programs, and probe performance does not identify the admissible velocity readout space. Shallow-stratum probe accuracy of 0.927 also coexists with generation accuracy of 0.646 (Section 3.4), suggesting that decodable information can remain unused. These limitations matter if the empirical failures are presented as a general computational barrier.

The benefit of Plan+Paint combines resolved instructions with additional rendering passes and changes to the editing procedure. Appendix C.6 describes the pipeline, but the results do not establish how much gain remains under a comparable total inference budget. This weakens the practical compute-allocation claim, although it does not remove the value of the controlled diagnosis.

Questions for authors

Which conclusions are intended as guarantees only for the theoretical constructions, and which are supported for the actual backbones? Please explain how the random-oracle setting relates to SynBool without assuming that explicit Boolean computations have the same oracle access restriction.

Can the existing runs be summarized in terms of planner tokens, renderer evaluations, and end-to-end cost per arm? A comparison at a shared inference budget would clarify the allocation claim.

What does the shallow probe–generation gap imply about the proposed bottleneck? Does the evidence favor unavailable answers, unused decodable information, or both in different depth regimes?

Reviewer_QLEzOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the execution gap in text-to-image generation: diffusion and flow matching models can render explicitly specified visual targets but often fail to determine what to generate from instructions requiring multiple reasoning steps.

The authors introduce SYNBOOL to separate instruction execution from visual rendering. On Boolean programs with 15 operations, FLUX achieves only 27.1% accuracy with unresolved instructions, compared with 80.7% when the correct answer is supplied and 96.7% with a minimal visual prompt. Increasing denoising steps does not effectively close this gap.

To explain these findings, the paper introduces semantic availability and analyzes three constraints: content, adaptive computation, and timing. A velocity-risk analysis under restricted semantic readouts shows that statistically complete inputs do not necessarily provide usable semantic information. A computational lower bound for random-oracle chains illustrates the limitations of substituting parallel computation for sequential dependencies. A controllability analysis shows that, within the proposed linear control model, later semantic disclosure increases the minimum correction energy required to reach a fixed target accuracy.

The proposed Plan + Paint strategy resolves discrete reasoning through an upstream autoregressive planner and distributes spatial execution across incremental rendering passes. It improves task accuracy over the compared approaches on SynGrid, GenEval, and T2I-ReasonBench.

Critical strengths
  1. The Boolean program task provides unambiguous ground-truth answers while keeping visual rendering simple. The substantial accuracy improvement when the correct answer is supplied indicates that a considerable share of failures arises from instruction execution, rather than an inability to render the requested colors.
  2. The paper directly tests whether additional denoising computation improves reasoning. Increasing sampling steps from 8 to 100 yields no consistent accuracy gains on complex SYNBOOL tasks. This provides a useful perspective for UMM research: reasoning scalability should be assessed by whether additional computation supports semantic state transitions, rather than merely by counting sampling steps or model forward passes.
Critical weaknesses
  1. Theorem 1 establishes velocity collapse under a stylized sign-product flow matching model with a restricted semantic readout class. However, the paper does not establish that practical DiT backbones satisfy these assumptions, leaving the connection to the observed generation failures insufficiently validated. Comparing velocity predictions under resolved and unresolved conditioning across denoising timesteps, potentially alongside causal interventions such as activation patching, would provide more direct evidence for the proposed mechanism.
  2. The SYNBOOL experiments increase the number of Boolean operations from 3 to 31, simultaneously changing program length, total computational work, and potentially sequential dependency depth. The resulting accuracy degradation therefore cannot be conclusively attributed to serial depth alone. A stronger controlled experiment would hold the operation count, prompt length, and gate distribution fixed while varying the dependency graph’s span. A complementary experiment could vary parallel work while keeping the span fixed. Both experiments should report generation and probing accuracy.
Questions for authors

Please address the questions and missing comparisons raised in the Critical Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.