acceptodds
Under review as a conference paper at ICLR 2027

Text-to-Image Models Need Less from Text Encoders Than You Think

Abstract

Text-to-image models rely on text prompts as their primary interface to human intent. Prompts are encoded by a text encoder into embeddings that condition the image generation process. Beyond individual token meanings, text embeddings encode contextual information across the full prompt, such as compositionality and attribute binding. However, whether image models actually exploit this richer information remains underexplored. Here, we address the question: Which aspects of text representation are essential for image generation? We show that text-to-image diffusion transformer-based models commonly rely only on two relatively straightforward aspects of text representations: (i) the merging of adjacent tokens into a word representation, for words spanning multiple tokens, and (ii) word order, which is imprinted by the positional embedding of the text-encoder. To show this, we construct a new text embedding that encodes only individual word meanings and order but lacks any contextual information about the full prompt. We find that this bag of position-tagged words representation is sufficient to successfully guide image generation, achieving visual quality and text fidelity that are on par with full text embedding-guided generation. This demonstrates that, contrary to common belief, text-to-image models often do not use the rich information encoded in the text embedding beyond individual word meanings and word order. Instead, the decoding of complex linguistic structures is performed by the image model itself.

Primary area:
generative models
Keywords:
Text-to-image generation, text encoders, text conditioning, diffusion models, Language-image models

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_DZr6OpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates which aspects of text representations are necessary for conditioning pretrained text-to-image models. The authors construct alternative embeddings by averaging contextual token representations across unrelated sentences. The BoT, BoW, and BoPTW variants progressively preserve token identity, constituent word identity, and absolute token position, while leaving the pretrained image generator unchanged.Experiments evaluate SD3, FLUX.1 Schnell, and FLUX.2 Klein-4B using VLM preferences, native GenEval scores, image-quality metrics, and a human study. Additional experiments examine long prompts and compare DiT with U-Net architectures. The central finding is that BoPTW preserves substantial generation capability in several settings, which the authors interpret as evidence that much of linguistic processing occurs within the image model.

Critical strengths

1.A relevant research question and a useful intervention. Distinguishing information encoded in text representations from information necessary for generation is important. The BoT–BoW–BoPTW progression provides an intuitive framework for studying word identity and positional information without modifying the pretrained generator. 2. Complementary empirical evaluations. The study includes multiple generators and prompt distributions, native GenEval scores, a human study, and explicit failure cases. These evaluations provide useful evidence beyond selected qualitative examples. 3. An interesting architectural comparison. The DiT–U-Net contrast is noteworthy. The synthetic experiment using the same frozen T5 encoder and approximately matched model sizes strengthens this observation beyond comparisons between independently developed systems.

Critical weaknesses
  1. Text-representation quality should be analyzed jointly with inference budget and training regime.My central concern is that final-image performance captures only one aspect of the value of a text representation. A representation may support satisfactory generation under a particular sampling configuration while still affecting how efficiently the generator reaches that result and how successfully it can be adapted to a different generation objective.This concern is motivated by EMF [1], which reports differences in few-step semantic preservation between BLIP3o-NEXT and SANA-1.5, and links textual representation properties to the behavior of MeanFlow training. EMF also discusses the possibility of trajectory correction under multi-step sampling. These findings do not directly invalidate BoPTW, but they motivate examining whether endpoint robustness conceals differences that become important under tighter inference budgets.The present paper does not report the actual sampling steps, schedulers, or guidance settings for its main experiments, nor a systematic analysis of how the full-versus-BoPTW gap changes with sampling budget. Importantly, FLUX.1 Schnell and the non-Base FLUX.2 Klein-4B are themselves few-step-distilled models, so it would be incorrect to claim that few-step systems are absent. The unresolved issue is the scope and budget dependence of the finding.A matched comparison across supported sampling budgets would substantially strengthen the analysis. Iterative compensation should be treated as a possible explanation to test, not an established cause. Even if BoPTW is sufficient at a particular budget, that does not establish comparable efficiency or suitability for alternative training regimes.

  2. Removing prompt-specific contextualization does not establish that powerful text encoders have limited value.The intervention retains averaged representations produced by large pretrained encoders. These vectors can preserve rich lexical semantics and representation geometry acquired during pretraining. Consequently, the experiments test whether prompt-specific contextualization can be replaced in a frozen pipeline; they do not directly test whether inexpensive word-level encoders would provide equally useful conditioning.More generally, the object of interest is the embedding actually supplied to the generator, which depends on both the encoder and the prompt. Prompt construction or rewriting can alter the information supplied to the model and how that information is represented. The current analysis does not establish how robust its conclusions are to semantically matched paraphrases, changes in instruction structure, or different levels of prompt detail. The degradation on DPG makes this boundary particularly relevant.The adoption of LLM-based conditioning in FLUX.2 and MLLM-based text encoding in HunyuanVideo [2] does not itself prove that larger encoders are necessary. It does, however, motivate distinguishing the benefits of contextualization from other benefits of pretrained representations and prompt processing. I would therefore narrow the practical implication from questioning the need for powerful encoders broadly to identifying circumstances in which their prompt-specific contextualization can be bypassed. Training a new simplified system is a reasonable future direction, not a prerequisite for publishing this analysis.

  3. The quantitative evidence does not support a broad “on par” characterization.The reported “non-inferiority rate” is a descriptive win-or-tie fraction, not a statistical non-inferiority test. Counting ties toward both methods is legitimate, but high values do not establish comparable performance, particularly because Section B.3 explicitly permits a tie when both images are poor.For FLUX.2 on GenEval, Table 6 reports 3.1% BoPTW wins, 63.8% ties, and 33.1% full-embedding wins under Gemma. Thus, a 66.9% win-or-tie rate coexists with a substantial preference imbalance. Table 4 provides stronger task-level evidence: overall GenEval decreases from 82.3 to 71.3, two-object performance from 92.7 to 58.9, and attribute binding from 66.8 to 49.5.The corresponding overall results are much more favorable for FLUX.1, changing from 70.4 to 70.7, while SD3 changes from 66.0 to 63.9. This supports a model-dependent conclusion rather than uniform parity. DPG win-or-tie rates of 41.5–57.0% further restrict the long-prompt claim. The abstract and conclusion should reflect these substantial differences.

  4. Behavioral sufficiency is not equivalent to non-use or a demonstrated mechanism.Successful generation after replacing full embeddings does not establish that the original model does not use contextual information when it is available. A generator can exploit contextualized embeddings while also being capable of reconstructing relationships from word identities and positions.Similarly, the progressive localization of attention in Figure 20 is consistent with internal semantic processing, but it does not by itself establish the claimed causal mechanism or show that positional tags are responsible for resolving the relationships. The paper’s repeated “do not use” and “proves” formulations are stronger than the evidence warrants. The findings more directly establish robustness to a particular intervention under particular conditions.

References [1].Zhao, Chenxi, et al. "Extending One-Step Image Generation from Class Labels to Text via Discriminative Text Representation." arXiv preprint arXiv:2604.18168 (2026) [2].Kong, Weijie, et al. "Hunyuanvideo: A systematic framework for large video generative models." arXiv preprint arXiv:2412.03603 (2024).

Questions for authors

1.What exact checkpoints, sampling steps, schedulers, guidance settings, seed-pairing procedures, and text-truncation policies were used in the main experiments?

2.Within each model’s supported sampling range, how does the performance gap between full and BoPTW conditioning change with the inference budget? Can the authors distinguish their inference-time findings from claims about the requirements of training or adapting efficient generators, as studied in EMF?

3.How stable are the findings across semantically matched paraphrases and controlled changes in prompt detail? Which benefits inherited from the original pretrained encoder remain in the averaged representations?

4.Can the authors provide prompt-level uncertainty estimates and absolute task-success comparisons, including results restricted to cases where full conditioning succeeds? How should “on par” be reconciled with the FLUX.2 results in Tables 4 and 6?

5.What evidence distinguishes “the generator can operate without prompt-specific contextualized embeddings” from “the generator does not use contextual information in full embeddings”? Could the mechanistic claims be qualified accordingly?

7 comments
AuthorsOpenReview ↗

Response to Reviewer DZr6 - part 1

The influence of text embeddings on training (W1)

We thank the reviewer for this insightful point. We agree that text representation quality can fundamentally shape a model's training regime. Please note that we trained two conditional generative models of comparable capacity (30M parameters): a convolutional U-Net and a diffusion Transformer (DiT). Both models receive representations from a frozen T5-XXL text encoder (see section B.3.1 in the Appendix) and found cohesion with the results in the main paper. Actual sampling steps, schedulers, or guidance settings (W1,Q1) In all experiments we use the default configuration of the image model. All image-generation configurations are kept identical across our comparisons, including the number of sampling steps (28 for SD3, 4 for FLUX.1 and FLUX.2), sampling scheduler (FlowMatchEulerDiscreteScheduler), CFG scale (0.0 for FLUX.1, 1.0 for FLUX.2 and 7.5 for SD.3). We added it to the paper (Table 9).

The intervention retains averaged representations produced by large pretrained encoders (W2)

We thank the reviewer for the opportunity to clarify this important distinction. The reviewer is correct that BoPTW embeddings preserve contextual statistics associated with a word appearing at a particular position. As the reviewer notes, these statistics are independent of the specific input prompt and do not encode the relationships between words appearing together in that particular prompt. Rather, they reflect general patterns learned by the pretrained text encoder across different contexts. However, this does not imply that word-level encoders cannot provide such conditioning. Constant position-dependent statistical patterns can be captured by fixed, precomputed embeddings for each word at each position, without requiring prompt-specific contextualization. Therefore, they could potentially be learned by a word-level encoder with a positional encoding mechanism, if it is trained together with the image model. In fact, the patterns do not even require an encoder; they can be captured, e.g. by a fixed precomputed dictionary for each position (similarly to how we construct our embeddings). We would like to emphasize that we do not advocate for these approaches as practical solutions. We provide these examples only to illustrate that supplying the image model with fixed per-position patterns does not necessitate the contextualization mechanism of current text encoders.

Acknowledgment of sensitivity limitations (W2)

To address the reviewer’s prompt-sensitivity concern, we conducted additional experiments using semantically matched paraphrases by randomly sampling 100 prompts from the MSCOCO validation dataset and generating five context-preserving variations for each using the FLUX.1 model. For instance, for the prompt “A group of stuffed animals and guinea pig sitting on a couch.”, the variations include “A fluffy guinea pig relaxing on a sofa alongside an assortment of colorful stuffed animals.”, “A cute guinea pig hanging out and cuddling with a collection of plush toys on a cozy couch.”, “Several plush animals and a real guinea pig lounging comfortably together on a living room couch, soft lighting.”, “A close-up shot of a live guinea pig sitting right in the middle of various stuffed animals on a fabric sofa.”, and “A comfortable couch occupied by a mix of stuffed animals and a guinea pig.” For each variation, we generated five images with different seeds using both the full embedding and the BoPTW embedding. Applying the evaluation protocol established in the paper we get a non-inferiority rate of 68%. Specifically, for 32% of the prompts the full embeddings are preferred, for 45% of the two embeddings are equally preferred (ties) and for 23% of the prompts the BoPTW embeddings are preferred. We also measured the absolute success rates using a VLM that is tasked to judge whether each individual image adheres to the text prompt. The success rates were 75% for the full embeddings and 72% for the BoPTW embeddings. Using the Wilson test, we get confidence intervals of [71.%, 78.3%] for the full embeddings and [68.28%, 75.44%] for the BoPTW embeddings.

AuthorsOpenReview ↗

Response to Reviewer DZr6 - part 2

Clarifying the "On Par" Claim (W3)

Our central finding is that, somewhat surprisingly, removing prompt-specific contextual information often preserves a large fraction of the image generation model's capabilities, even for prompts involving complex linguistic relationships. In the paper, we primarily evaluate this phenomenon using VLM-based comparisons across MS-COCO, DrawBench, and GenEval, measuring the frequency with which generations using BoPTW are not judged inferior to those using full embeddings. We use the term “on par” specifically to refer to the VLM-based evaluation across all three datasets (MS-COCO, DrawBench, and GenEval). This conclusion applies consistently across all experiments using this evaluation protocol. In particular, our results demonstrate near-complete sufficiency across most prompts for FLUX.1 and SD3 across all evaluated datasets, as well as for FLUX.2 across most evaluated prompt categories. We would also like to emphasize that, even when considering the GenEval benchmark (Table 5), FLUX.1 and SD3 achieve success rates with contextless embeddings that are close to those obtained with full contextual embeddings (63.9 vs. 66.0 and 70.7 vs. 70.4, respectively). For FLUX.2, the performance gap is larger (71.3 vs. 82.3), particularly for two-object generation and attribute binding. Nevertheless, BoPTW retains substantial capabilities even in this setting, achieving strong results in single-object generation, counting, and color fidelity single-object generation, counting, color, and position despite the removal of prompt-specific contextual information. We also acknowledge that the performance gap becomes more pronounced for long, complex prompts, as demonstrated by the DPG results. These findings indicate that contextual information can be important, particularly for certain models and tasks. To better reflect these findings and avoid overly broad claims, we have revised the abstract and main text, replacing blanket statements that the methods are “on par” with the more nuanced phrasing “usually on par.”

The reported “non-inferiority rate” (W3)

We focus on the question of when contextless embeddings are sufficient to generate images that adhere to the prompt. From this perspective, the non-inferiority rate is an appropriate metric, as it measures the proportion of prompts for which full contextual embeddings are not preferred, indicating that contextless embeddings are sufficient to achieve comparable prompt adherence. We would like to emphasize that a-priori, one would not expect the contextless embeddings to outperform the full embeddings on any prompt. In other words, one would expect the non-inferiority rate, which combines wins and ties, to be dominated by ties. The fact that in practice we observe a non negligible proportion of wins, is a surprising observation in its own right.

Behavioral sufficiency is not equivalent to non-use or a demonstrated mechanism (W4)

We agree with the reviewer on this subtlety. Our experiments demonstrate that replacing contextualized text embeddings with BoPTW representations preserves much of the generation performance across the evaluated models and benchmarks. This shows that rich, prompt-specific contextualization is not necessary to achieve this level of performance under the evaluated conditions. We agree, however, that this does not establish that the image generator does not exploit contextual information when it is available, nor does it reveal how the generator processes the different representations internally. In particular, the generator may leverage contextual information from full embeddings while also being capable of exploiting word identities and positions when contextualization is removed. Establishing whether, and how, the generator uses contextual information internally would require a separate investigation beyond the scope of our behavioral comparisons. To address this concern, we have revised the manuscript to distinguish between the sufficiency of BoPTW representations for generation and claims about the internal use or processing of contextual information. We have also moderated our wording to avoid implying that contextual information is never used by the image generator. Instead of the phrase "models do not use" we wrote "models are robust to the removal of prompt-specific context" and instead of "proves" we wrote "demonstrates that DiTs can reconstruct relational structure without context".

AuthorsOpenReview ↗

Response to Reviewer DZr6 - part 3

Interpretation of attention maps (W4)

We agree that the progressive localization observed in Figure 20 is consistent with internal semantic processing but does not establish a causal mechanism. Nevertheless, it illustrates that the image is progressively constructed across the denoising timesteps, even when conditioning on contextless embeddings. Accordingly, we replaced the word "proves" with the more cautious formulation "demonstrates that DiTs can reconstruct relational structure without prompt-specific contextualization."

Stability across semantically matched paraphrases and controlled changes in prompt detail (Q3)

We thank the reviewer for raising these points. To address this concern, we conducted additional experiments using semantically matched paraphrases by randomly sampling 100 prompts from the MSCOCO validation dataset and generating five context-preserving variations for each using FLUX.1 model. For instance, given the prompt “A group of stuffed animals and guinea pig sitting on a couch.”, the variations include “A fluffy guinea pig relaxing on a sofa alongside an assortment of colorful stuffed animals.”, “A cute guinea pig hanging out and cuddling with a collection of plush toys on a cozy couch.”, “Several plush animals and a real guinea pig lounging comfortably together on a living room couch, soft lighting.”, “A close-up shot of a live guinea pig sitting right in the middle of various stuffed animals on a fabric sofa.”, and “A comfortable couch occupied by a mix of stuffed animals and a guinea pig.” Then, we generated five images per variation across seeds using both the full embedding and the BoPTW contextless embeddings, applying the evaluation protocol established in the paper we get a non-inferiority rate of 68%.Specifically, in 32% of the prompts, the full embedding is preferred, at 45% in the prompts, equal preferences (ties) and for 23% of the prompts the contextless embedding is preferred. We also measured the absolute task-success. Using a VLM to judge whether the image is adhered to the text prompt or not, we got absolute task-success 75% for the full embeddings and 72% for the contextless embeddings. Using the Wilson test, we get confidence intervals of [71.%, 78.3%] for the full embeddings and [68.28%, 75.44%] for the BoPTW contextless embeddings.

Which benefits inherited from the original pretrained encoder remain in the averaged representations? (Q3)

The averaged representations remain in, or close to, the representation manifold learned by the pretrained text encoder, thereby preserving its pretrained semantic structure but nothing related to the context of the word or token. Please kindly note that once a token's embedding is taken from one prompt and transplanted into a different prompt, it is already "context-free" with respect to the new prompt. Namely, it no longer carries the context of the prompt it was originally embedded in. For example, if we take the embedding of the token "cat" from the prompt "The cat is black" and insert it in place of the token "cat" in the prompt "The cat and the dog", that embedding is now decoupled from its original context ("The cat is black") and carries no information about the new prompt's context ("The cat and the dog") either. And naturally, this holds in our case as well: we take the embedding of each token from at least 10 (in many cases from much more, above hundreds) different prompts (with their own distinct context) and use it within a different prompt that has an entirely different context from those we used to extract the embeddings.

Can the authors provide prompt-level uncertainty estimates and absolute task-success comparisons (Q3)

We thank the reviewer for this suggestion. We measured both prompt-level uncertainty and absolute success rates across all datasets and models. We used Gemma-3 as a judge to evaluate whether each generated image adhered to its corresponding text prompt. We added the meta-prompt used for this evaluation to the manuscript. For each model and dataset, we separately measured the number of successful and number of unsuccessful generations according to the VLM, using the same number of seeds per prompt as in the paper (5 for GenEval and DrawBech, 1 for MSCOCO). We then calculated the success and failure rates and reported the corresponding 95% confidence intervals using the Wilson method. See Figure 10. We can see that the gap between the contextless and the full embedding is on average ~7% in SD3, ~5% in FLUX.1 and ~13% in FLUX.2.

Reviewer_rFanOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks which aspects of text embeddings text-to-image models actually use. It constructs contextless embeddings by averaging each token embedding over unrelated sentences, and analyzes two aspects in particular, the merging of adjacent tokens into words and word order. The resulting BoPTW embedding, which keeps only word identity and position, is reported to be non-inferior to the full embedding in many cases on SD3, FLUX.1, and FLUX.2, from which the authors conclude that the image model itself decodes most of the linguistic structure.

Critical strengths

The contextless prompt is the core idea of the paper and, in my view, both its main strength and its main weakness. It offers a simple drop-in probe that requires no change to the image model and gives an interesting perspective on the division of labor between the text encoder and the image model.

Critical weaknesses

W1. Unclear problem with rich prompts and unsupported implications The paper does not make clear what problem rich prompts actually have, which makes it difficult to appreciate the significance of the findings. The end of the introduction and the conclusion suggest that smaller text encoders or training on word-level embeddings may suffice, but no experiment supports these implications. In practice, constructing BoPTW is more expensive than using the full embedding, since it requires encoding all prompts in CC3M and MSCOCO with the full text encoder and generating sentences with Claude at inference time for multi-token words (Sec. 4.1, B.2). To support its claims, the paper needs a concrete problem definition, for example in terms of efficiency, or by showing that concepts omitted under the full context are preserved by a representation such as BoPTW.

W2. Analysis design and Figure 1 Figure 1 presents curated success cases, while failure cases appear only in the appendix (Figs. 12 and 17). The progression from BoT to BoPTW shows the effect of adding word-level and positional information, but there is no intermediate step between BoPTW and the full embedding, such as partial context that allows only neighboring words. Such an analysis would show more directly how much context the image model actually needs.

W3. The "on par" claim is not consistently supported Full embeddings remain better than BoPTW both qualitatively and quantitatively, and the gap is substantial in several settings. On GenEval (Table 4), FLUX.2 drops from 82.3 to 71.3 overall, from 92.7 to 58.9 on two objects, and from 66.8 to 49.5 on attribute binding. On long prompts (DPG), the non-inferiority of BoPTW is 47.2%, 57.0%, and 41.5% for SD3, FLUX.1, and FLUX.2. The largest gap appears for FLUX.2, the most recent model with an LLM text encoder, which runs counter to the general claim in the title.

Minor BoPTW averages over sentences in which the word appears at the same position, so contextual statistics typical of that position may remain in the embedding, and the extent to which it is truly contextless should be discussed. In addition, Fig. 8 shows that fewer than half of the tokens are localized with zero error, whereas Sec. A.1 states that the vast majority are.

Questions for authors

Please refer to the "Critical Weaknesses" section.

In particular, I personally believe that clearly resolving the concerns about the contextless representation would best demonstrate the novelty of this paper.

5 comments
AuthorsOpenReview ↗

Response to Reviewer rFan21 - part 1

Problem settings (W1)

We thank the reviewer for raising this concern, which suggests that we should clarify the motivation and scope of our work. Our goal is not to identify a limitation of rich prompts or to propose BoPTW as a more computationally efficient alternative to existing text encoders. Rather, we investigate a fundamental question about text-to-image models: which aspects of the rich representations produced by text encoders are actually utilized by the downstream image generation model? In particular, we aim to understand whether the image model relies on contextual information already computed by the text encoder, or whether it can infer complex linguistic relationships from much simpler representations. We therefore do not propose BoPTW as a direct replacement for the text encoder. Rather, we introduce it as a diagnostic tool for gaining insights into the underlying process. By removing prompt-specific contextual information while preserving word identity and position, we can investigate which information is necessary for generating images with pretrained models. Our findings demonstrate that, in many cases, replacing complex, context-dependent embeddings with BoPTW can be sufficient, even when dealing with long and sophisticated prompts. This observation is, in itself, an interesting insight for the research community, particularly given the common assumption that scaling up text encoders is an inevitable direction of progress. The implications of these findings for the future design of text encoders could be substantial. Depending on the implementation, several possibilities can be envisioned, including smaller text encoders, partial text encoders, hybrid architectures combining partial text encoders with BoPTW, or alternative embedding mechanisms derived from BoPTW. We leave the exploration of these specific architectural choices to future work.

Computational Efficiency (W1)

We agree with the reviewer that constructing BoPTW in our current experimental setup is computationally expensive. However, this cost arises from our methodology for studying existing pretrained models, rather than from an inherent requirement of the representations themselves. In particular, BoPTW assigns a fixed embedding to each word at each position, independently of the surrounding context. This raises the possibility of designing future architectures that directly learn such embeddings during training, without requiring a large contextual text encoder or the additional computations used in our experiments to construct BoPTW. We emphasize that we do not claim to demonstrate computational savings in the present work. Rather, our findings provide empirical motivation for exploring these potentially more efficient architectures, whose effectiveness remains to be established.

Analysis design and Figure 1 and intermediate cases (W2)

We thank the reviewer for this comment! To address this concern, we conduct an additional experiment in which we average the embeddings of consecutive pairs and triplets of words with overlapping of a single word. For example, given the prompt “A photo of a black refrigerator”, we construct the following representations: Pairs: We learn averaged embeddings for the consecutive word pairs: “A photo”, “photo of”, “of a”, “a black”,and “black refrigerator”. Triplets: We learn averaged embeddings for the consecutive word triplets: “A photo of”, “photo of a”, “of a black” and “a black refrigerator”. We construct the corresponding dictionaries using 10 different prompts, ensuring that each pair or triplet appears at the same position across prompts. We follow the same protocol for constructing the embeddings (Figure 3) and for evaluation (Section 4.1) as described in the paper.

Results for the GenEval dataset:

For the Bag-of-Position-Tagged-Pair-Words representation, we obtain:

FLUX.1: 75.1% non-inferiority rate vs 73.6% with BoPTW.

SD3: 72.1% non-inferiority rate vs 68.6% with BoPTW.

For the Bag-of-Position-Tagged-Triplet-Words representation, we obtain:

FLUX.1: 76.6% non-inferiority rate vs 73.6% with BoPTW.

SD3: 74.9% non-inferiority rate vs 68.6% with BoPTW.

These results show a slight improvement over BoPTW. However, this gap is still smaller than the gap between BoT and BoPTW (which is +21.6% for FLUX.1 and +20.5% for SD3). This reinforces our observation that the image model benefits significantly from extracting from the text embedding word positions and context for tokens belonging to the same word. The marginal benefit of context involving consecutive words is smaller.

AuthorsOpenReview ↗

Response to Reviewer rFan21 - part 2

Qualifying the "On Par" Claim (W3)

Our central finding is that, somewhat surprisingly, removing prompt-specific contextual information often preserves a large fraction of the image generation model's capabilities, even for prompts involving complex linguistic relationships. In the paper, we primarily evaluate this phenomenon using VLM-based comparisons across MS-COCO, DrawBench, and GenEval, measuring the frequency with which generations using BoPTW are not judged inferior to those using full embeddings. We use the term “on par” specifically to refer to the VLM-based evaluation across all three datasets (MS-COCO, DrawBench, and GenEval). This conclusion applies consistently across all experiments using this evaluation protocol. In particular, our results demonstrate near-complete sufficiency across most prompts for FLUX.1 and SD3 across all evaluated datasets, as well as for FLUX.2 across most evaluated prompt categories. We would also like to emphasize that, even when considering the GenEval benchmark (Table 5), FLUX.1 and SD3 achieve success rates with contextless embeddings that are close to those obtained with full contextual embeddings (63.9 vs. 66.0 and 70.7 vs. 70.4, respectively). For FLUX.2, the performance gap is larger (71.3 vs. 82.3), particularly for two-object generation and attribute binding. Nevertheless, BoPTW retains substantial capabilities even in this setting, achieving strong results in single-object generation, counting, and color fidelity single-object generation, counting, color, and position despite the removal of prompt-specific contextual information. We also acknowledge that the performance gap becomes more pronounced for long, complex prompts, as demonstrated by the DPG results. These findings indicate that contextual information can be important, particularly for certain models and tasks. To better reflect these findings and avoid overly broad claims, we have revised the abstract and main text, replacing blanket statements that the methods are “on par” with the more nuanced phrasing “usually on par.”

Remaining context or BoPTW averages over sentences

We thank the reviewer for the opportunity to clarify this important distinction. The reviewer is correct that BoPTW embeddings may preserve contextual statistics associated with a word appearing at a particular position. However, these statistics are independent of the specific input prompt and do not encode the relationships between words appearing together in that prompt. Rather, they reflect general patterns learned by the pretrained text encoder across different contexts. Importantly, these statistical patterns can be captured by fixed, precomputed embeddings for each word at each position, without requiring prompt-specific contextualization. Our central finding is that image models can often generate images that adhere to complex prompts using these fixed representations. In fact, we find that BoPTW outperforms BoW, suggesting that the image model benefits from the additional position-dependent information, potentially including general linguistic patterns captured by the text encoder. Crucially, these patterns are fixed and do not depend on the specific input prompt. Thus, such information could be learned once and reused across prompts, without requiring a large contextual text encoder to process each input prompt. This further motivates our broader question of how much linguistic processing truly needs to be performed by the text encoder. We will clarify this distinction in the paper, noting that our use of the term "contextless" refers specifically to the absence of prompt-specific contextualization, rather than the complete absence of contextual statistics.

Fig. 8 vs. Sec. A.1

We thank the reviewer for catching this phrasing mismatch. In Fig. 8, 44.2% of the tokens are localized with strict zero error (pos=0), while 31.4% fall within an error tolerance of 1 position (which Sec. A.1 loosely called "the vast majority"). We have corrected Sec. A.1 to state the exact zero-error rate and 1 rate to match Fig. 8 precisely.

Dear Reviewer rFan, as the CVPR deadline has passed, I guess our hands are a little bit untied now. Given that we only have very little time, can you please respond to the author's comments as soon as possible?

Comments on the paper

6 comments
AuthorsOpenReview ↗

Response to Reviewer s3sX - part 1

The reported “non-inferiority rate” (W1)

We focus on the question of when contextless embeddings are sufficient to generate images that adhere to the prompt. From this perspective, the non-inferiority rate is an appropriate metric, as it measures the proportion of prompts for which full contextual embeddings are not preferred, indicating that contextless embeddings are sufficient to achieve comparable prompt adherence. We would like to emphasize that a-priori, one would not expect the contextless embeddings to outperform the full embeddings on any prompt. In other words, one would expect the non-inferiority rate, which combines wins and ties, to be dominated by ties. The fact that in practice we observe a non-negligible proportion of wins, is a surprising observation in its own right.

The non-inferiority-rate masks a strong directional preference for full embeddings (W1)

We added a table reporting the proportion of cases that contextless embeddings are preferred or tied, compared to those in which full contextual embeddings are preferred or tied. In most cases the values are close (see Table 4 in the revised paper).

Prompt-level uncertainty and absolute success rates (W1)

We thank the reviewer for this suggestion. We measured both prompt-level uncertainty and absolute success rates across all datasets and models. We used Gemma-3 as a judge to evaluate whether each generated image adhered to its corresponding text prompt. We added the meta-prompt used for this evaluation to the manuscript. For each model and dataset, we separately measured the number of successful and number of unsuccessful generations according to the VLM, using the same number of seeds per prompt as in the paper (5 for GenEval and DrawBech, 1 for MSCOCO). We then calculated the success and failure rates and reported the corresponding 95% confidence intervals using the Wilson method. See Figure 10. We can see that the gap between the contextless and the full embedding is on average ~7% in SD3, ~5% in FLUX.1 and ~13% in FLUX.2.

Qualifying the "On Par" Claim (W1)

Our central finding is that, somewhat surprisingly, removing prompt-specific contextual information often preserves a large fraction of the image generation model's capabilities, even for prompts involving complex linguistic relationships. In the paper, we primarily evaluate this phenomenon using VLM-based comparisons across MS-COCO, DrawBench, and GenEval, measuring the frequency with which generations using BoPTW are not judged inferior to those using full embeddings. We use the term “on par” specifically to refer to the VLM-based evaluation across all three datasets (MS-COCO, DrawBench, and GenEval). This conclusion applies consistently across all experiments using this evaluation protocol. In particular, our results demonstrate near-complete sufficiency across most prompts for FLUX.1 and SD3 across all evaluated datasets, as well as for FLUX.2 across most evaluated prompt categories. We would also like to emphasize that, even when considering the GenEval benchmark (Table 5), FLUX.1 and SD3 achieve success rates with contextless embeddings that are close to those obtained with full contextual embeddings (63.9 vs. 66.0 and 70.7 vs. 70.4, respectively). For FLUX.2, the performance gap is larger (71.3 vs. 82.3), particularly for two-object generation and attribute binding. Nevertheless, BoPTW retains substantial capabilities even in this setting, achieving strong results in single-object generation, counting, and color fidelity single-object generation, counting, color, and position despite the removal of prompt-specific contextual information. We also acknowledge that the performance gap becomes more pronounced for long, complex prompts, as demonstrated by the DPG results. These findings indicate that contextual information can be important, particularly for certain models and tasks. To better reflect these findings and avoid overly broad claims, we have revised the abstract and main text, replacing blanket statements that the methods are “on par” with the more nuanced phrasing “usually on par.”

Refining Mechanistic Claims (W2)

We agree with the reviewer on this subtlety and rephrased the manuscript accordingly. Instead of the phrase "models do not use" we wrote "models are robust to the removal of prompt-specific context" and instead of "proves" we wrote "demonstrates that DiTs can reconstruct relational structure without context".

Absolute Metrics for the Synthetic DiT vs. U-Net Control (W2)

To address the request for absolute performance data alongside relative preference rates, we added the absolute relational classification accuracy for both the full-model baseline and the BoPTW variant on our held-out synthetic test set (DiT: Full = 86.4%, BoPTW = 81.6%; U-Net: Full = 42.3%, BoPTW = 35.9%).

AuthorsOpenReview ↗

Response to Reviewer s3sX - part 2

Technical Implementation Details (W2)

Generally, we are using the suggestion from huggingface at each model. Following the reviewer’s comment, we added an explicit methodology subsection clarifying all pipeline mechanics:

  • Tokenizer-Level Verification: We bridge word-level Claude carrier prompts and absolute tokenizer indices using a strict verification filter that discards any candidate sentence where the target word's subtoken start index does not precisely match the required position.

  • Special Tokens: Regarding the padding tokens, as stated in the paper, “we pad the remaining positions with the embedding of an empty string to reach the maximum sequence length.” We apply the same principle to the start tokens, using the start-token embeddings of an empty string. We clarified this in the paper.

  • Pooled CLIP Outputs: As we wrote in the paper, “we apply pooling to the per-token contextless CLIP embeddings in the same manner used for the full embeddings“. The implementation is part of the FLUX HF code and includes max pooling.

  • pooled_output = last_hidden_state[ torch.arange(last_hidden_state.shape[0], device=last_hidden_state.device), input_ids.to(dtype=torch.int, device=last_hidden_state.device).argmax(dim=-1), ]

  • Generator-Side Positionals: All internal DiT positional embeddings (such as RoPE on text token indices) remain completely unmodified.

Quantitative Sensitivity Analysis for Figure 19 (W2)

Please note that Figure 19 examines the effect of the number of embeddings used to construct the contextless representations. As illustrated by the examples, averaging a single embedding from a different context or averaging 10 embeddings yields visually similar results. To further address this concern, we report the CLIP scores for different numbers of averaged embeddings (M) across SD3, FLUX.1, and FLUX.2. Increasing M from 1 to 3 yields a slight improvement across all three models, while further increasing M does not provide additional gains.

MSD3FLUX.1FLUX.2
133.233.532.6
233.233.532.6
333.433.632.7
433.433.632.7
533.433.632.7
633.433.632.7
733.433.632.7
833.433.632.7
933.433.632.7
1033.433.632.7

We added this table to the updated revision (Table 7).

Problem settings (W2)

We thank the reviewer for raising this concern, which suggests that we should better clarify the motivation and scope of our work. Our goal is not to identify a limitation of rich prompts or to propose BoPTW as a more computationally efficient alternative to existing text encoders. Rather, we investigate a fundamental question about text-to-image models: which aspects of the rich representations produced by text encoders are actually utilized by the downstream image generation model? In particular, we aim to understand whether the image model relies on contextual information already computed by the text encoder, or whether it can infer complex linguistic relationships from much simpler representations. We therefore do not propose BoPTW as a direct replacement for the text encoder. Rather, we introduce it as a diagnostic tool for gaining insights into the underlying process. By removing prompt-specific contextual information while preserving word identity and position, we can investigate which information is necessary for generating images with pretrained models. Our findings demonstrate that, in many cases, replacing complex, context-dependent embeddings with BoPTW can be sufficient, even when dealing with long and sophisticated prompts. This observation is, in itself, an interesting insight for the research community, particularly given the common assumption that scaling up text encoders is an inevitable direction of progress. The implications of these findings for the future design of text encoders could be substantial. Depending on the implementation, several possibilities can be envisioned, including smaller text encoders, partial text encoders, hybrid architectures combining partial text encoders with BoPTW dictionaries precomputed for each token location, or alternative embedding mechanisms derived from BoPTW. We leave the exploration of these specific architectural choices to future work.

AuthorsOpenReview ↗

Thank you to the reviewers for their valuable feedback! We have uploaded a new revised version addressing their comments and suggestions.

We thank the reviewers for their thoughtful and constructive feedback. We have uploaded a revised version of the manuscript that addresses the reviewers' comments and incorporates the additional clarifications and experiments. We would be happy to address any additional concerns or questions the reviewers may have.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.