acceptodds
Withdrawn from ICLR 2027

A Language-Informed Loss for Visual Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

End-to-end visual generation with pixel-space loss has recently achieved competitive performance without the need for learned image tokenizers. In this work, we study whether language-informed supervision can provide a richer training signal than pixel loss for visual generation. We introduce **LILIE** (pronounced "lily"), a **L**anguage-**I**nformed and **L**ossless **I**mage **E**ncoder. LILIE is an invertible image transform trained such that a vision-language model can recover captions from noised features. The encoder uses a normalizing flow transformer trained jointly with a vision-language model via next token prediction. We then use the encoder to extract a simple language-informed loss for text-to-image generation. We train JiT models with LILIE loss, reducing gFD by 16 points (68.61 52.73) against a pixel-only baseline on the GPIC dataset. Our code, encoders, and generative models are included in the Supplementary.

Primary area:
generative models
Keywords:
generative modeling, representation learning, text-to-image, visual generation, image generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_F3qyOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LILIE (Language-Informed and Lossless Image Encoder), an invertible image transform trained so that a vision-language model can recover captions from noise-corrupted features. Built on an improved TARFlow architecture and jointly trained with a LoRA-finetuned SmolVLM-256M via next-token prediction, LILIE learns a language-supervised yet lossless feature space; the VLM is then discarded and the frozen encoder is used to extract an additional L2 loss in LILIE-space for text-to-image generation. On GPIC at 256×256 with JiT-T2I models, LILIE loss alone reduces gFD from 68.61 to 55.78, and combining Pixel and LILIE losses reaches 52.73 for the 1.1B generator—a 16-point gain over the pixel-only baseline—with consistent improvements at 700M. Ablations show that scaling the encoder (1.5M→32M) and its training data (10M→100M) both improve quality, while LILIE surpasses the GPIC-compliant LPIPS baseline as a compact, standalone loss. Limitations are limited to one corpus, resolution and generator family, dependence on captions and a single small VLM, and encoder sizes far below DINOv2/SigLIP 2.

Critical strengths

++ The paper is well written and organized.

++ The contribution is interesting and useful.

++ The experimental results demonstrate the loss function is effective.

Critical weaknesses

We do not have many additional comments, and the following two points are the ones I am particularly concerned about. Moreover, the workload required for revisions and supplementary experiments will be extremely substantial. For this reason, I only put forward these two suggestions:

-- I am particularly concerned about the impact of the number of network layers or the network depth in the LILIE module on the loss function of this paradigm, and how it compares with the conventional perceptual loss. I am not quite sure whether this performance improvement comes from the model's stronger feature learning capability. In addition, I also pay close attention to its generalization ability and zero-shot learning performance.

-- This idea is well-motivated: you have proposed a simple loss function paradigm that remains to be fully validated for its effectiveness, which can be regarded as a semantic-enhanced version of the conventional perceptual loss. Nevertheless, the authors still need to sufficiently verify the effectiveness of this loss paradigm across diverse downstream tasks, to facilitate its broader adoption and further research in subsequent works.

Questions for authors

See Weakness

1 comment
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for their valuable time and feedback. Below, we address both suggestions with motivating experiments.

W1: Impact of the architecture’s feature learning capability.

We thank the reviewer for raising this concern. We perform a model scaling study (Table 2), in which we vary model width and model depth in reducing the size of the LILIE encoder. We notice that even with a small 1.5M parameter encoder consisting of only 4 transformer layers, we observe a large improvement in JiT-T2I gFD (101.66 →\to 84.66 for the 1.1B generator at 200K steps). These results show that encoder size affects generation quality and that smaller encoders still improve on pixel-only training. We additionally verify that LILIE’s gain is attributed to informative captions. We hold the encoder architecture fixed and repeat LILIE training with random tokens in place of the ground-truth captions.

Experiment setup. During LILIE encoder training, we replace each caption’s tokens with independent tokens sampled uniformly from the SmolVLM-256M vocabulary observed in GPIC training captions. We then train a JiT-T2I-XL model for 390,600 steps using Pixel+LILIE supervision.

Results. The random-token experiment achieves 82.06 gFD, compared with 66.00 using ground-truth captions. It also fails to improve on pixel-only training (79.94). These results show that informative language is a critical ingredient for the gain from LILIE, not only the encoder architecture.

Generator lossLILIE training captionsgFD ↓
Pixel—79.94
Pixel+LILIERandom tokens82.06
Pixel+LILIEGround-truth captions66.00

W2: Downstream tasks.

We thank the reviewer for this suggestion. The reviewer’s description of LILIE as a semantic enhancement of a perceptual loss motivates the following experiment: when two image edits have the same pixel error, is LILIE more sensitive to semantic errors?

Experimental setup. We start with 10,000 randomly selected GPIC validation images at 256 × 256. We make 2 copies of each image. In one, we replace randomly selected 16 × 16 patches with gray. In the other, we brighten, darken, or desaturate the entire image. We adjust the strength of tone change and masking to 24 dB PSNR relative to the original per sample. For each image, we compute the distance in each semantic space from the original image to its masked copy and to its tone-adjusted copy. We then report the percentage of images for which the masked copy is farther. Masking corrupts semantics of an image more than slight tone changes, so a semantic feature space loss should assign higher distances to masked errors. We report mean ± two standard errors over images.

Results. LILIE assigned a larger distance to the masked image than to the brightened image in 99.42% of pairs and to the masked image than to the darkened image in 97.28% of pairs. These were the highest rates among the four methods for both comparisons. For desaturation, DINOv2 had the highest rate (86.77%), followed by LILIE (73.53%).

Larger distance measured (↑\uparrow)LILIELPIPSDINOv2SigLIP 2
Masked over Brightened99.42 ± 0.1597.92 ± 0.2998.35 ± 0.2596.85 ± 0.35
Masked over Darkened97.28 ± 0.3392.70 ± 0.5293.08 ± 0.5189.92 ± 0.60
Masked over Desaturated73.53 ± 0.8863.23 ± 0.9686.77 ± 0.6853.65 ± 1.00
Reviewer_sgrmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LILIE, a language-informed generation loss based on an invertible image encoder trained with caption supervision from a pretrained vision-language model. The encoder is frozen and used to measure feature-space differences between generated and target images.

Critical strengths
  1. The proposed method improves generation performance on GPIC, demonstrating the potential of combining caption supervision with invertible representations.
  2. The paper is well organized, with clear explanations, detailed architectural descriptions, and informative visualizations that support reproducibility.
Critical weaknesses
  1. The methodological novelty requires further clarification, as caption-supervised representation learning and feature perturbation have been explored in previous studies. The connection between the encoder design and the claimed contribution of a generation loss also needs clearer justification.
  2. The benefits of invertibility are not sufficiently validated, as the ablation studies do not isolate its contribution from caption supervision and pretrained VLM knowledge.
  3. The experiments do not directly demonstrate whether the proposed loss distinguishes semantic differences under comparable pixel-space errors, despite this being a key motivation.
  4. The evaluation is limited to GPIC and the JiT-T2I generator family, leaving the generalizability of the proposed loss insufficiently established.
Questions for authors
  1. How does the proposed loss affect training time and memory usage compared with the pixel-only baseline?
  2. Can the effects of invertibility and pretrained VLM knowledge be evaluated separately?
  3. Does LILIE distinguish perturbations with similar pixel errors but different semantic impacts compared with existing perceptual losses?
  4. Can the proposed loss be evaluated on additional datasets or generator architectures?
2 comments
AuthorsOpenReview ↗

Rebuttal by Authors [1/2]

W1: The methodological novelty requires further clarification.

We thank the reviewer for raising this concern. The reviewer is correct in that caption-supervised representation learning has been proposed in prior work, notably in CapPa [A] and LocCa [B]. We note that SigLIP 2 [C] is trained with a LocCa objective and fails to work as a standalone loss (Table 1). This motivates our choice of a TARFlow backbone instead of a ViT as invertibility guarantees that LILIE loss is 0 only when the predicted image exactly matches the target image, allowing LILIE loss to work as a standalone loss. Our method also uses noise as a corruption mechanism to make caption-relevant information robust to corruption. We provide a table below of representative language-informed or trained-with-noise methods that have been used for visual generation to illustrate how LILIE differs. Our main contribution in this work is a language-informed loss that works as a standalone loss for end-to-end visual generation.

MethodLanguage-informedTrained with noiseTwo-stage generationEnd-to-end generation
TexTok [D]✓✓
UniTok [E]✓✓
DeTok [F]✓✓
FlowMo [G]✓✓
LILIE✓✓✓

[A] Tschannen et al. Image Captioners Are Scalable Vision Learners Too. NeurIPS, 2023.

[B] Wan et al. LocCa: Visual Pretraining with Location-aware Captioners. NeurIPS, 2024.

[C] Tschannen et al. SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. arXiv, 2025.

[D] Zha et al. Language-Guided Image Tokenization for Generation. CVPR, 2025.

[E] Ma et al. UniTok: A Unified Tokenizer for Visual Generation and Understanding. NeurIPS, 2025.

[F] Yang et al. Latent Denoising Makes Good Tokenizers. ICLR, 2026.

[G] Sargent et al. Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization. ICCV, 2025.

W2 + Q2: Isolating the effects of invertibility, caption supervision, and pretrained VLM knowledge.

We thank the reviewer for raising this question and propose three experiments to isolate the effects of invertibility, caption supervision, and pretrained VLM knowledge.

Invertibility

Experimental setup. We preserve the LILIE training configuration but replace our 32.3M parameter TARFlow encoder with a 32.5M parameter ViT. We then train a JiT-T2I-XL with Pixel+LILIE-ViT loss for 200K steps. We sweep λ∈2,0.5,0.1,0.01\lambda \in \\{2,0.5,0.1,0.01\\}.

Results. Our best result for LILIE-ViT is at λ=0.01\lambda =0.01, attaining a gFD of 115.59, underperforming both Pixel and Pixel+LILIE at 114.59 and 91.79, respectively. We additionally observe that gFD monotonically increases as λ\lambda increases.

λ0 (pixel)0.010.10.52
gFD ↓114.59115.59131.76275.46531.86

This result suggests that the standard vision transformer does not work well when used as a plug-and-play replacement for TARFlow in LILIE training. This also provides evidence that LILIE does not only benefit from the transformer’s strong learning capability.

Caption supervision

Experiment setup. During LILIE encoder training, we replace each caption’s tokens with independent tokens sampled uniformly from the SmolVLM-256M vocabulary observed in GPIC training captions. We then train a JiT-T2I-XL model for 390,600 steps using Pixel+LILIE supervision.

Results. The random-token experiment achieves 82.06 gFD, compared with 66.00 using ground-truth captions. It also fails to improve on pixel-only training (79.94). These results show that informative language is a critical ingredient for the gain from LILIE.

Generator lossLILIE training captionsgFD ↓
Pixel-only—79.94
Pixel+LILIERandom tokens82.06
Pixel+LILIEGround-truth captions66.00

Pretrained VLM knowledge

Experimental setup. We replace SmolVLM-256M with Qwen3.5-0.8B during LILIE encoder training, retaining the LILIE-S encoder architecture and GPIC caption supervision. We then train JiT-T2I-g for 390,600 steps using either standalone LILIE or Pixel+LILIE supervision.

Results. Replacing SmolVLM-256M with Qwen3.5-0.8B improves standalone gFD from 55.78 to 52.40. Adding pixel loss improves gFD with SmolVLM, while standalone LILIE gives the best result with Qwen.

JiT-T2I-g gFD (↓\downarrow)

VLM used to train LILIEStandalone LILIEPixel+LILIE
SmolVLM-256M55.7852.73
Qwen3.5-0.8B52.4053.90

We hope this series of experiments addresses the reviewer’s question regarding the isolation of effects in LILIE training.

AuthorsOpenReview ↗

Rebuttal by Authors [2/2]

W3 + Q3: Does LILIE distinguish perturbations with similar pixel errors but different semantic impacts compared with existing perceptual losses?

We thank the reviewer for suggesting this comparison. We test whether LILIE penalizes the removal of image content more than a global change in brightness or saturation when both edits have the same pixel error. Here we consider brightness and saturation changes to be less semantic than the removal of image content.

Experimental setup. We start with 10,000 randomly selected GPIC validation images at 256 × 256. We make 2 copies of each image. In one, we replace randomly selected 16 × 16 patches with gray. In the other, we brighten, darken, or desaturate the entire image. We adjust the strength of tone change and masking to 24 dB PSNR relative to the original per sample. For each image, we compute the distance in each semantic space from the original image to its masked copy and to its tone-adjusted copy. We then report the percentage of images for which the masked copy is farther. Masking damages semantics of an image more than slight tone changes, so a semantic feature space loss should assign higher distances to masked errors. We report mean ± two standard errors over images.

Results. LILIE assigned a larger distance to the masked image than to the brightened image in 99.42% of pairs and to the masked image than to the darkened image in 97.28% of pairs. These were the highest rates among the four methods for both comparisons. For desaturation, DINOv2 had the highest rate (86.77%), followed by LILIE (73.53%).

Larger distance measured (↑\uparrow)LILIELPIPSDINOv2SigLIP 2
Masked over Brightened99.42 ± 0.1597.92 ± 0.2998.35 ± 0.2596.85 ± 0.35
Masked over Darkened97.28 ± 0.3392.70 ± 0.5293.08 ± 0.5189.92 ± 0.60
Masked over Desaturated73.53 ± 0.8863.23 ± 0.9686.77 ± 0.6853.65 ± 1.00

W4 + Q4: Can the proposed loss be evaluated on additional datasets or generator architectures?

We use GPIC as it provides a large, permissively-licensed, safe, and stable benchmark for academic research on visual generation. We address the reviewer’s concern by performing a pretraining experiment using minit2i [H] on the CC12M dataset [I].

Experimental setup. We follow the released setup from minit2i and pretrain a minit2i-B/32 model at 512x512 resolution on the CC12M dataset for 250K steps. We pretrain both a Pixel-only baseline as well as Pixel+LILIE.

Results. Pixel+LILIE improve on the Pixel-only baseline by 4 points on MSCOCO-30K FID (29.253 -> 24.847), 3 points on GenEval [J] (42.55 -> 45.53), and 2 points on DPGBench [K] (74.58 -> 76.58). Despite our best efforts, we are unable to reproduce the exact Pixel-only results from the minit2i article with the released JAX configuration.

Training lossMSCOCO-30K FID ↓GenEval ↑DPGBench ↑
Pixel-only29.25342.5574.58
Pixel+LILIE24.84745.5376.58

[H] Wang et al. MiniT2I: A Minimalist Baseline for Text-to-Image Generation. Technical blog, 2026.

[I] Changpinyo et al. Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts. CVPR, 2021.

[J] Ghosh et al. GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment. NeurIPS, 2023.

[K] Hu et al. ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment. arXiv, 2024.

Q1: How does the proposed loss affect training time and memory usage compared with the pixel-only baseline?

We thank the reviewer for raising this important question. LILIE introduces minimal training overhead (less than 5% overhead in training time and less than 2% in memory usage) when used to train the downstream generator. When comparing wall-clock training time on a TPU v4-32 between Pixel-only and Pixel+LILIE JiT-T2I, we observe a 4.1% slowdown for Pixel+LILIE at the 700M parameter generator size and 3.4% slowdown at the 1.1B parameter generator size. When comparing memory usage, we observe a 1.6% increase for Pixel+LILIE at the 700M parameter generator size and a 1.1% increase at the 1.1B parameter generator size.

Reviewer_ZUMwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a language-informed loss (LILIE) for visual generation. More specifically, the authors first train an invertible encoder (a TARFlow) jointly with a LoRA-finetuned vision-language model (VLM), such that the VLM can caption the encoder’s noise-corrupted output. The VLM is then discarded, and the frozen encoder is used to compute an L2 loss in its feature space for training pixel-space diffusion models (JiT-T2I), either alone or combined with pixel loss. Experimental results demonstrate that LILIE improves over the pixel-only baseline in reported experimental setting and performs on par with other feature-space losses when combined with pixel loss.

Critical strengths
  • Using the LILIE loss alone, without a pixel-loss term, outperforms the pixel-only baseline (gFD 68.61 → 55.78 for the 1.1B model). This finding shows that a learned invertible transform can provide a valid training signal, whereas LPIPS, DINOv2, and SigLIP 2 losses collapse when used alone.
Critical weaknesses

Feature-space losses have been explored in many areas. The two main claims of this paper are (1) a language-informed loss (as written in the title) and (2) a lossless encoder. However, neither of them is well supported.

  • Language-informed loss
    • According to the main results, the language-informed loss is not clearly better than DINO or LPIPS, when combined with pixel loss. Besides, only a single VLM has been tested, without ablations across different models or sizes.
    • SigLIP 2 is also a kind of “language-informed” loss. This means that the gain of the proposed method may come not from language, but from the architecture or the training objective.
  • Lossless encoder
    • The lossless encoder design choice is not supported by experiments. To show the necessity of the lossless module, there should be a comparison with a non-lossless module of similar size (e.g., an MLP). Then the authors could show that their model clearly outperforms it in the motivated scenario.
    • The main recommended setting combines LILIE with pixel loss. In this setting, the pixel term already guarantees the correct minimizer, so invertibility of the encoder is not required for well-posed training, as the Pixel+LPIPS/DINOv2/SigLIP 2 results show. Invertibility is only essential for the standalone LILIE loss, which underperforms the combined loss.
Questions for authors
  • Could you report the results of replacing TARFlow with a similar-sized MLP?
  • The authors added LoRA to the VLM during training, but I couldn’t find a clear reason for this. Could you explain why LoRA is required and report the results without LoRA?
  • In Table 4, the authors state that LILIE used 100M training images. I think this does not count the images used in SmolVLM training. Could you clarify this?
  • All experiments are at 256x256 resolution. It is unclear whether the gains persist at higher resolution. Could you report the results of 512x512 or 1024x1024 resolution experiments?
  • Could you report qualitative results of LILIE only (without pixel loss)?
8 comments
AuthorsOpenReview ↗

Rebuttal by Authors [1/2]

We thank the reviewer for their valuable time and feedback. Below, we address all the concerns with requested experiments.

W1: LILIE loss is not clearly better than DINOv2 or LPIPS when combined with pixel loss.

The reviewer is correct. When combined with pixel loss, LILIE does not consistently outperform existing perceptual losses. In fact, the aim of this work is to construct a standalone space with language supervision to supervise end-to-end visual generation. As one can see in Table 1, LILIE outperforms pixel-only, DINOv2-only, SigLIP 2-only, and LPIPS-only losses, but we also observe that all of these methods benefit from combining pixel loss.

Nevertheless, to address combined performance, we note that LILIE can complement existing perceptual supervision. We run the following experiment.

Experimental setup. We pretrain JiT-T2I-g for 390,600 steps on GPIC-Full. We combine pixel, LILIE, and LPIPS supervision weighted by LL = LPixel+2LLILIE+0.5LLPIPSL_\text{Pixel} + 2 L_\text{LILIE} + 0.5 L_\text{LPIPS}.

Results. Pixel+LILIE+LPIPS achieves the best performance yet, achieving a gFD of 45.27 compared to the existing best setup of Pixel+DINOv2 at 47.88.

W1: Only a single VLM has been tested.

We thank the reviewer for their suggestion. We select Qwen3.5-0.8B to investigate whether LILIE benefits from a larger VLM with stronger visual understanding capabilities. In addition to increasing model scale from 256M to 0.8B parameters, Qwen obtains higher benchmark scores than SmolVLM-256M on MMMU (47.4 versus 28.3) and MMStar (55.9 versus 34.6) [A, B].

Experimental setup. We replace SmolVLM-256M with Qwen3.5-0.8B during LILIE encoder training, retaining the LILIE-S encoder architecture and GPIC caption supervision. We then train JiT-T2I-g for 390,600 steps using either standalone LILIE or Pixel+LILIE supervision.

Results. Replacing SmolVLM-256M with Qwen3.5-0.8B improves standalone gFD from 55.78 to 52.40. Adding pixel loss improves gFD with SmolVLM, while standalone LILIE gives the best result with Qwen.

JiT-T2I-g gFD (lower is better)

VLM used to train LILIEStandalone LILIEPixel+LILIE
SmolVLM-256M55.7852.73
Qwen3.5-0.8B52.4053.90

[A] Qwen Team. Qwen3.5-0.8B model card. Hugging Face, 2026.

[B] HuggingFaceTB. SmolVLM-256M-Instruct model card. Hugging Face, 2025.

W1: SigLIP 2 is language-informed. LILIE’s gain may not come from language.

Yes, the reviewer is correct in that SigLIP 2 is also language-informed. We highlight a key difference: SigLIP 2 is trained with captioning, image-text alignment, and self-supervised objectives, whereas LILIE is only trained with a captioning objective. The reviewer’s statement that “gain of the proposed method may come not from language” raises a valid concern, and we perform an experiment where we replace ground-truth captions in LILIE training with random tokens to isolate the effect of informative captions.

Experiment setup. During LILIE encoder training, we replace each caption’s tokens with independent tokens sampled uniformly from the SmolVLM-256M vocabulary observed in GPIC training captions. We then train a JiT-T2I-XL model for 390,600 steps using Pixel+LILIE supervision.

Results. The random-token experiment achieves 82.06 gFD, compared with 66.00 using ground-truth captions. It also fails to improve on pixel-only training (79.94). These results show that informative language is a critical ingredient for LILIE’s performance.

Generator lossLILIE training captionsgFD ↓
Pixel-only—79.94
Pixel+LILIERandom tokens82.06
Pixel+LILIEGround-truth captions66.00

W2 + Q1: Could you report the results of replacing TARFlow with a similar-sized MLP?

The reviewer is correct in that we did not ablate the lossless encoder. Our experiments show that SigLIP, LPIPS, and DINO all failed as standalone end-to-end generation losses. These observations motivate our use of a TARFlow architecture, whose invertibility guarantees the preservation of image information, to build a standalone loss for end-to-end visual generation. We thank the reviewer for this suggestion and detail the experiment below.

AuthorsOpenReview ↗

Rebuttal by Authors [2/2]

Experimental setup. We preserve the LILIE training configuration but replace our 32.3M parameter TARFlow encoder with a 32.5M parameter ViT. We then train a JiT-T2I-XL with Pixel+LILIE-ViT loss for 200K steps. We sweep λ∈2,0.5,0.1,0.01\lambda \in \\{ 2, 0.5, 0.1, 0.01 \\}.

Results. Our best result for LILIE-ViT is at λ=0.01\lambda = 0.01, attaining a gFD of 115.59, underperforming both Pixel and Pixel+LILIE at 114.59 and 91.79, respectively. We additionally observe that gFD monotonically increases as λ\lambda increases.

λ0 (pixel)0.010.10.52
gFD ↓114.59115.59131.76275.46531.86

This result suggests that the standard vision transformer does not work well when used as a plug-and-play replacement for TARFlow in LILIE training. This also provides evidence that LILIE does not only benefit from the transformer’s strong learning capability.

W2 + Q1: Invertibility is not required for well-posed training.

The reviewer is correct. In our optimal setting, which is Pixel+LILIE, invertibility is not required due to the Pixel term. We reiterate that the goal of this work is to construct a simple standalone space in which to supervise end-to-end visual generation. In addition, in the Qwen3.5-0.8B experiment, we find that removing pixel loss (LILIE only) yields the optimal result.

Q2: Could you explain why LoRA is required and report the results without LoRA?

We thank the reviewer for this question. We provide a brief motivation for LoRA in Appendix A and mention that LoRA allows the VLM to adapt to previously out-of-domain values. We validate this choice empirically by ablating a frozen VLM during LILIE training.

Experimental setup. We train a LILIE-S encoder with a frozen VLM (no LoRA), keeping the rest of the configuration unchanged. We then train a JiT-T2I-XL with Pixel+LILIE loss for 200K steps.

Results. Freezing the VLM yields a gFD of 201.71, compared with 91.79 using LoRA.

This substantial degradation illustrates the advantage of allowing the VLM to adapt during LILIE training.

Q3: LILIE training images.

The reviewer is correct, the reported 100M training images for LILIE does not include the images used in SmolVLM training. Our TARFlow encoder is initialized from scratch, while SmolVLM-256M is initialized from pretrained weights trained on The Cauldron and Docmatix datasets [B].

[B] HuggingFaceTB. SmolVLM-256M-Instruct model card. Hugging Face, 2025.

Q4: Could you report the results of 512x512 or 1024x1024 resolution experiments?

The reviewer is correct in that we only perform experiments at the 256x256 resolution. We note that foundational visual generation research is predominantly done at the 256x256 resolution (e.g., DiT [C], JiT [D], MAR [E], RCG [F], RAE [G], REPA [H]). However, the reviewer raises a valuable concern, so we perform 512x512 pretraining experiments following minit2i [I]. This new setup is intended to provide evidence towards generalization across model architectures and datasets in addition to resolution.

Experimental setup. We follow the released setup from minit2i and pretrain a minit2i-B/32 model at 512x512 resolution on the CC12M dataset for 250K steps. We pretrain both a Pixel-only baseline as well as Pixel+LILIE.

Results. Pixel+LILIE improve on the Pixel-only baseline by 4 points on MSCOCO-30K FID (29.253 -> 24.847), 3 points on GenEval [J] (42.55 -> 45.53), and 2 points on DPGBench [K] (74.58 -> 76.58). Despite our best efforts, we are unable to reproduce the exact Pixel-only results from the minit2i article with the released JAX configuration.

Training lossMSCOCO-30K FID ↓GenEval ↑DPGBench ↑
Pixel29.25342.5574.58
Pixel+LILIE24.84745.5376.58

[C] Peebles et al. Scalable diffusion models with transformers. ICCV 2023.

[D] Li et al. Back to basics: Let denoising generative models denoise. CVPR 2026.

[E] Li et al. Autoregressive image generation without vector quantization. NeurIPS 2024.

[F] Li et al. Return of unconditional generation. NeurIPS 2024.

[G] Zheng et al. Diffusion transformers with representation autoencoders. arXiv 2025.

[H] Yu et al. Representation alignment for generation. ICLR 2025.

[I] Wang et al. MiniT2I: A Minimalist Baseline for Text-to-Image Generation. Technical blog, 2026.

[J] Ghosh et al. GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment. NeurIPS, 2023.

[K] Hu et al. ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment. arXiv, 2024.

Q5: Could you report qualitative results of LILIE only (without pixel loss)?

We provide qualitative results of standalone LILIE in Appendix G.

Dear Authors,

Thank you for the rebuttal and the additional experiments. After reading it carefully, most of my concerns remain. I detail them below.

1. LILIE is not clearly better than existing losses, and all experiments are far from convergence

The rebuttal acknowledges that, when combined with pixel loss, LILIE does not consistently outperform existing perceptual losses. In addition, all models in the paper and the rebuttal are trained for at most one epoch, and many ablations for only 200K steps. The best gFD (52.73 in the paper, 45.27 in the rebuttal) remains far from the real-data reference on GPIC (7.44 for Test-50K [1]). Therefore, all comparisons are made in an early, unconverged regime. Prior work shows that gains from auxiliary feature-based supervision can be large early in training but plateau or even reverse later [2]. Without results closer to convergence, it is unclear whether the gain from LILIE persists, or whether LILIE only accelerates early training. Could you report training curves, or results with substantially longer training, for Pixel and Pixel+LILIE?

Regarding the new Pixel+LILIE+LPIPS result (45.27), it is compared with Pixel+DINOv2 (47.88), but Pixel+DINOv2+LPIPS is not reported. The improvement may therefore come from combining more losses rather than from LILIE.

2. Only a single VLM has been tested

The same convergence concern applies to the Qwen3.5-0.8B experiment. Moreover, with Qwen3.5-0.8B, Pixel+LILIE (53.90) is worse than with SmolVLM-256M (52.73) and worse than standalone LILIE (52.40). Why does a stronger VLM degrade the recommended Pixel+LILIE setting? Without variance across seeds, it is also unclear whether these differences are meaningful.

3. Language-informed

The rebuttal agrees that SigLIP 2 is also language-informed. The random-token control does not isolate the role of language, since random captions provide essentially no meaningful training signal; it shows only that some meaningful supervision is needed. The title and the paper emphasize that the loss is "language-informed", but it has not been shown that language, rather than any semantic supervision, is the source of the gain. I think the current framing overstates this contribution. A control using a non-language objective of similar strength (e.g., class labels or a self-supervised target) on the same TARFlow encoder is needed.

4. TARFlow vs. ViT

I do not find this ablation convincing. LILIE-ViT is worse than pixel-only at every λ and diverges at λ=2 (gFD 531.86). However, in the paper, DINOv2 and SigLIP 2, which are both ViTs, improve over pixel-only when combined with pixel loss. A ViT feature-space loss that is harmful at every weight is therefore unlikely to reflect a property of the architecture; it more likely indicates a training issue, such as feature-norm growth to overcome the added noise. Could you report the feature norms of both encoders, and results with normalized features or smaller λ? As it stands, this ablation does not support the need for an invertible encoder.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.