- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LILIE (Language-Informed and Lossless Image Encoder), an invertible image transform trained so that a vision-language model can recover captions from noise-corrupted features. Built on an improved TARFlow architecture and jointly trained with a LoRA-finetuned SmolVLM-256M via next-token prediction, LILIE learns a language-supervised yet lossless feature space; the VLM is then discarded and the frozen encoder is used to extract an additional L2 loss in LILIE-space for text-to-image generation. On GPIC at 256×256 with JiT-T2I models, LILIE loss alone reduces gFD from 68.61 to 55.78, and combining Pixel and LILIE losses reaches 52.73 for the 1.1B generator—a 16-point gain over the pixel-only baseline—with consistent improvements at 700M. Ablations show that scaling the encoder (1.5M→32M) and its training data (10M→100M) both improve quality, while LILIE surpasses the GPIC-compliant LPIPS baseline as a compact, standalone loss. Limitations are limited to one corpus, resolution and generator family, dependence on captions and a single small VLM, and encoder sizes far below DINOv2/SigLIP 2.
Critical strengths
++ The paper is well written and organized.
++ The contribution is interesting and useful.
++ The experimental results demonstrate the loss function is effective.
Critical weaknesses
We do not have many additional comments, and the following two points are the ones I am particularly concerned about. Moreover, the workload required for revisions and supplementary experiments will be extremely substantial. For this reason, I only put forward these two suggestions:
-- I am particularly concerned about the impact of the number of network layers or the network depth in the LILIE module on the loss function of this paradigm, and how it compares with the conventional perceptual loss. I am not quite sure whether this performance improvement comes from the model's stronger feature learning capability. In addition, I also pay close attention to its generalization ability and zero-shot learning performance.
-- This idea is well-motivated: you have proposed a simple loss function paradigm that remains to be fully validated for its effectiveness, which can be regarded as a semantic-enhanced version of the conventional perceptual loss. Nevertheless, the authors still need to sufficiently verify the effectiveness of this loss paradigm across diverse downstream tasks, to facilitate its broader adoption and further research in subsequent works.
Questions for authors
See Weakness
Rebuttal by Authors
We thank the reviewer for their valuable time and feedback. Below, we address both suggestions with motivating experiments.
We thank the reviewer for raising this concern. We perform a model scaling study (Table 2), in which we vary model width and model depth in reducing the size of the LILIE encoder. We notice that even with a small 1.5M parameter encoder consisting of only 4 transformer layers, we observe a large improvement in JiT-T2I gFD (101.66 → 84.66 for the 1.1B generator at 200K steps). These results show that encoder size affects generation quality and that smaller encoders still improve on pixel-only training. We additionally verify that LILIE’s gain is attributed to informative captions. We hold the encoder architecture fixed and repeat LILIE training with random tokens in place of the ground-truth captions.
Experiment setup. During LILIE encoder training, we replace each caption’s tokens with independent tokens sampled uniformly from the SmolVLM-256M vocabulary observed in GPIC training captions. We then train a JiT-T2I-XL model for 390,600 steps using Pixel+LILIE supervision.
Results. The random-token experiment achieves 82.06 gFD, compared with 66.00 using ground-truth captions. It also fails to improve on pixel-only training (79.94). These results show that informative language is a critical ingredient for the gain from LILIE, not only the encoder architecture.
We thank the reviewer for this suggestion. The reviewer’s description of LILIE as a semantic enhancement of a perceptual loss motivates the following experiment: when two image edits have the same pixel error, is LILIE more sensitive to semantic errors?
Experimental setup. We start with 10,000 randomly selected GPIC validation images at 256 × 256. We make 2 copies of each image. In one, we replace randomly selected 16 × 16 patches with gray. In the other, we brighten, darken, or desaturate the entire image. We adjust the strength of tone change and masking to 24 dB PSNR relative to the original per sample. For each image, we compute the distance in each semantic space from the original image to its masked copy and to its tone-adjusted copy. We then report the percentage of images for which the masked copy is farther. Masking corrupts semantics of an image more than slight tone changes, so a semantic feature space loss should assign higher distances to masked errors. We report mean ± two standard errors over images.
Results. LILIE assigned a larger distance to the masked image than to the brightened image in 99.42% of pairs and to the masked image than to the darkened image in 97.28% of pairs. These were the highest rates among the four methods for both comparisons. For desaturation, DINOv2 had the highest rate (86.77%), followed by LILIE (73.53%).