acceptodds
Withdrawn from ICLR 2027

GESSO: An Aesthetic Perceptual Loss for Pixel Space Text-to-Image Pretraining

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Aesthetic quality is conventionally added through data filtering, curation, preference finetuning, or reward optimization. Here, we ask whether a perceptual loss can supply aesthetic signal during pretraining. We study two design choices not previously examined in text-to-image pretraining. First, a system-level study of ten ViT-B encoders shows that language-supervised encoders perform best, while the self-supervised default, DINOv2, is last or tied for last on aesthetics, prompt alignment and fidelity. Swapping in SigLIP gives a stronger perceptual recipe. Second, how an aesthetic scorer enters the loss matters: with the scorer fixed, distilling it into a reference-anchored, per-patch distance improves held-out aesthetic metrics, whereas maximizing its score does not. We introduce GESSO, an aesthetic perceptual loss for pretraining: a lightweight projector over frozen SigLIP features maps a reference image and the model output to a distance that tracks the difference in their aesthetic scores, so aesthetic signal reaches every pretraining image rather than only a small curated set afterwards, at 0.8% extra step time. Added to the SigLIP recipe, GESSO improves every human-preference metric and is preferred by every VLM judge across three model scales. At B/32 the full recipe improves HPSv2.1 by 10.2% and reduces FID by 12.4% relative to the PixelGen recipe; GESSO alone contributes 4.3% and 6.0%. Prompt alignment is similar, and the gains persist after supervised finetuning.

Primary area:
generative models
Keywords:
text-to-image generation, pixel-space diffusion, perceptual loss, pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SLN5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies aesthetic supervision for pixel-space text-to-image pretraining. The authors first compare ten perceptual encoders, find that language-supervised encoders perform well under a shared recipe, and replace DINOv2 with SigLIP. They then introduce GESSO, which distills absolute teacher score gaps between reference images and their one-step reconstructions into a per-patch cosine distance produced by a lightweight projector on frozen SigLIP features. After training, the projector is frozen and its distance becomes an additional generator training loss. The authors report improvements in automated preference metrics and FID at three model scales, supported by three matched B/32 training repeats, VLM pairwise judgments, and complete-pipeline results after supervised finetuning.

Critical strengths

S1. The method provides a practical improvement that is straightforward to integrate. In a pipeline already using a SigLIP perceptual loss, GESSO reuses feature computation, adds approximately 0.8% to training step time, and incurs no inference overhead. This gives it practical value as a training recipe (Section 3.3, Table 5).

S2. The gains over a stronger baseline are consistent and have some evidence of repeatability. In Table 3, GESSO improves B/32 HPSv2.1 from 21.12 to 22.03 and reduces FID from 10.25 to 9.63 relative to PixelGen-SigLIP. Trends across scales and automated scorers, together with matched branches from three independent B/32 training runs, make me confident that the complete recipe's gains are not solely due to the encoder replacement or generation seeds (Tables 3–4 and 10).

Critical weaknesses

W1. Attribution of the central methodological contribution remains insufficient. In Table 2, the scalar alternatives backpropagate through the original MUSIQ network, whereas the per-patch alternative backpropagates through a trained SigLIP projector, changing the backbone, distillation procedure, and loss formulation simultaneously. Better performance of the complete recipe therefore does not establish that the advantage comes from the reference-anchored per-patch distance rather than transferring the teacher signal into a more optimizable representation. A random projector does not replace a scalar student trained with the same teacher supervision. In addition, the degradation-only control removes other projector training terms, leaving aesthetic supervision insufficiently separated from general reconstruction-quality supervision(Sections 4.3 and 4.9, Tables 2 and 11, Appendix B.3).

W2. The evaluation design limits confidence in generalization and the interpretation of preference gains. Appendix A.1.5 states that the teacher, switch-on step, and weight were selected on the reported benchmarks, without clearly establishing whether configuration selection and final evaluation used independent samples. Separately, VLM win rates include only pairs judged consistently after swapping image order, without complete win, loss, and exclusion counts. The judging prompt also prioritizes caption adherence and realism, so these results represent overall preference on that subset. Agreement across automated metrics is valuable, but extrapolation to human aesthetic preference should remain limited without direct human validation(Section 4.7, Appendices A.1.5 and B.6).

Questions for authors

Q1. Regarding W1: Are results available for scalar-score or global-distance students sharing the SigLIP backbone and teacher supervision, or degradation-supervision controls that preserve the auxiliary losses? If not, which existing evidence separately supports the independent roles of per-patch structure and aesthetic supervision? Please delimit the conclusions supported by the current experiments.

Q2. Regarding W2: Which prompts, samples, and metrics were used to select the teacher, switch-on step, and weight? Which final results were entirely excluded from configuration selection?

Q3. Regarding W2: Please provide each VLM's win, loss, and order-inconsistency counts for GESSO versus PixelGen-SigLIP. Do the conclusions hold when inconsistent judgments are treated as ties? Is human preference agreement already available for outputs from the models in this paper?

Reviewer_hRQKOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether aesthetic information can be incorporated into pixel-space text-to-image diffusion pretraining through a perceptual loss. The authors first compare ten pretrained ViT-B visual encoders and find that language-supervised encoders, particularly SigLIP, provide stronger perceptual supervision than DINOv2 in their experimental setting.

The authors then introduce GESSO, an aesthetic-aware perceptual loss that uses a lightweight projector on frozen SigLIP features. The projector is trained to align patch-level feature distances between reference and reconstructed images with aesthetic score differences estimated by HPSv3. This reference-anchored formulation is intended to preserve aesthetic-related properties during reconstruction rather than directly maximize aesthetic scores.

Experiments across three model scales demonstrate improvements in automated human-preference metrics and FID, while maintaining comparable prompt alignment. The paper further evaluates the contributions of different encoders, loss formulations, and aesthetic teachers through ablation studies, supervised fine-tuning experiments, and VLM-based pairwise comparisons.

Critical strengths

1. Systematic investigation of perceptual supervision.

The paper provides a useful empirical comparison of ten pretrained ViT-B encoders for perceptual supervision in pixel-space text-to-image generation (Table 1). The finding that language-supervised encoders, particularly SigLIP, outperform DINOv2 under the evaluated setting offers practical guidance for selecting perceptual-loss backbones. This is meaningful because the encoder choice has a substantial impact on generation performance.

2. Consistent empirical improvements with limited computational overhead.

The proposed GESSO loss consistently improves automated human-preference metrics and FID over PixelGen-SigLIP across three model scales (Tables 3–4). The authors also provide ablation studies examining alternative loss formulations and projector configurations, along with repeated training runs supporting the consistency of the improvements. Furthermore, GESSO introduces only approximately 0.8% additional training-step time over PixelGen-SigLIP and no inference overhead (Table 5), making it a potentially practical addition to existing pretraining pipelines.

Critical weaknesses

1. Insufficient evidence for aesthetic-specific improvements.

The central claim of this paper concerns aesthetic quality, yet the evaluation relies primarily on automated preference metrics (HPSv2.1, LAION-aesthetic, PickScore, and ImageReward), FID, and VLM-based judgments. While informative, these metrics can reflect multiple aspects of image quality, including semantic alignment, realism, and general visual preference, making it difficult to attribute the improvements specifically to aesthetic quality. In particular, the VLM evaluation explicitly prioritizes caption adherence and realism over aesthetics (Appendix B.6). More importantly, no blinded human evaluation is provided to validate whether the generated images are actually perceived as more aesthetically appealing. Given that aesthetic improvement is the central claim of this paper, a controlled human evaluation that separately assesses aesthetic appeal is essential to substantiate this contribution.

The qualitative evidence is also limited. Figure 1 presents only GESSO-generated samples without baseline comparisons, while Appendix Figure 6 provides side-by-side comparisons for only two selected prompts after supervised fine-tuning. Consequently, despite consistent quantitative improvements, the paper does not convincingly demonstrate the perceptual significance of its central aesthetic-specific contribution.

2. Limited methodological novelty and unclear problem formulation.

The proposed approach largely builds upon established perceptual-loss frameworks by combining a pretrained vision encoder, an aesthetic preference model, and a learned feature-space distance. While the systematic encoder comparison is informative, replacing DINOv2 with SigLIP is primarily an empirical backbone selection rather than a substantial algorithmic contribution.

The reference-anchored per-patch distance is the main methodological addition. However, the experiments in Table 2 do not fully isolate whether its improvements arise from aesthetic-specific supervision, patch-level localization, or stronger learned perceptual regularization. Although the untrained-projector ablation partially addresses this issue, more controlled comparisons with existing learned perceptual distances are needed to establish the necessity of the proposed formulation.

More fundamentally, the relationship between the proposed optimization objective and the claimed aesthetic improvements requires further clarification. As explained in Section 4.9, GESSO penalizes deviations from reference images according to their aesthetic impact rather than directly maximizing aesthetic scores. While this formulation may help preserve aesthetically important properties during training, the paper provides limited direct evidence explaining when and how such preservation translates into improved aesthetic quality at the model level, particularly for training images with relatively low aesthetic quality.

Furthermore, although GESSO consistently improves automated metrics, its additional gains over PixelGen-SigLIP are relatively modest, and the perceptual significance of these improvements is not sufficiently demonstrated through controlled qualitative comparisons. Given the limited algorithmic novelty, stronger evidence distinguishing GESSO from conventional learned perceptual regularization would help establish its unique contribution and practical significance.

Considering all the concerns and weaknesses above, my overall assessment leans toward rejection. My recommendation could change if the authors provide convincing aesthetic-specific validation and/or stronger evidence establishing the distinct contribution of GESSO beyond general perceptual regularization.

Questions for authors

Q1. Aesthetic-specific evaluation and human preference

Given that aesthetic quality is the central focus of this paper, could the authors provide more direct evidence that GESSO improves aesthetic appeal independently of semantic alignment and realism?

The current evaluation relies primarily on automated preference models, which may reflect multiple aspects of image quality. Although HPSv3 and several evaluation predictors are trained using human ratings or preferences, improvements in their scores do not necessarily correspond to proportional improvements in human-perceived aesthetic quality.

In particular, the VLM evaluation in Appendix B.6 explicitly prioritizes caption adherence and realism over aesthetics. Since the evaluation protocol records the criterion for each judgment, could the authors report the preference results separately for aesthetic quality, realism, and caption adherence, along with the proportion of undecided pairs?

Most importantly, are any blinded human preference evaluations available that specifically assess aesthetic appeal? Such evidence would materially strengthen the paper's central claim.

Q2. Qualitative significance of the improvements

The reported quantitative improvements are consistent, but the perceptual differences between the generated images are not sufficiently clear from the provided examples.

Figure 1 presents only GESSO-generated images, while Appendix Figure 6 includes controlled comparisons for only two selected prompts after supervised fine-tuning.

Could the authors provide more extensive side-by-side comparisons using identical prompts and generation seeds, particularly before supervised fine-tuning, where the main quantitative improvements are reported?

It would be especially helpful to identify which aesthetic properties are improved, such as composition, color harmony, lighting, or visual details, and whether these changes are consistently perceptible across different types of images.

Q3. Methodological novelty and the role of aesthetic-specific supervision

The proposed GESSO loss combines a pretrained vision encoder, an aesthetic preference model, and a learned feature-space distance. Similar components have been extensively investigated in existing perceptual-loss and learned perceptual-metric frameworks.

Could the authors clarify what fundamentally distinguishes GESSO from established learned perceptual distances and aesthetic-aware feature-learning approaches?

Table 2 demonstrates that the per-patch formulation outperforms scalar reward maximization and scalar score-gap matching. However, these comparisons also change the feature representation and loss parameterization.

Could the authors provide a more controlled comparison between global and patch-level learned distances using the same backbone, projector capacity, and aesthetic teacher? Such a comparison would help isolate the contribution of the proposed patch-level formulation.

Furthermore, Appendix A.1.5 shows that an untrained projector already improves performance, while the trained aesthetic projector provides additional gains. Although this ablation supports the usefulness of teacher-based supervision, it remains unclear whether the additional improvements are specific to aesthetic information or could be achieved by learning a general perceptual distance.

Could the authors compare GESSO against a projector trained using a non-aesthetic perceptual similarity objective under matched conditions? This would help establish whether aesthetic-specific supervision is essential to the observed improvements.

Q4. Aesthetic preservation versus aesthetic enhancement

Section 4.9 explains that GESSO pulls generated predictions toward their corresponding reference images rather than directly maximizing the teacher's aesthetic score. The loss therefore encourages the preservation of aesthetic-related properties in the training data.

While the authors provide an intuitive motivation for this formulation, the relationship between reference-level aesthetic preservation and model-level aesthetic enhancement would benefit from stronger empirical validation.

In particular, could the authors investigate how the effectiveness of GESSO varies with the aesthetic quality of the reference images?

Since GESSO penalizes deviations from each reference regardless of the direction of the aesthetic score difference, does the method provide comparable benefits for low- and high-aesthetic-quality training samples?

Could the authors also clarify under what conditions preserving aesthetic-related properties of individual reference images leads to improved aesthetic quality in generated samples?

Such evidence would help distinguish the practical role of GESSO as an aesthetic-aware reconstruction regularizer from its claimed benefits for aesthetic enhancement.

Q5. Contribution of encoder selection and practical significance

The encoder comparison is informative, but the relative contribution of GESSO should be distinguished from the improvement obtained by selecting a stronger perceptual encoder.

For example, at H/32, the HPSv2.1 improvement from PixelGen to PixelGen-SigLIP is +1.49, whereas adding GESSO provides a further +0.67 (Table 4). Thus, at this model scale, most of the total improvement over PixelGen comes from encoder replacement rather than GESSO itself.

Could the authors clarify the distinct practical benefits of GESSO beyond selecting a stronger perceptual encoder, particularly in terms of perceptible aesthetic improvements?

Additionally, Table 1 uses a perceptual-loss weight tuned for DINOv2 rather than independently optimizing the weight for each encoder. How robust is the encoder ranking to differences in loss-weight selection and pretraining configurations?

Finally, while the additional training-step overhead of GESSO is small, its projector requires a separate offline training procedure. Could the authors report the overall computational cost of this procedure, including the data-generation and projector-training stages?

Could the authors also discuss whether the observed benefits are expected to generalize beyond the evaluated pretraining setup?

These clarifications would help assess whether GESSO offers a broadly useful methodological contribution beyond the specific experimental configuration.

Reviewer_DXoaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates aesthetic improvement in pixel-space text-to-image generation through perceptual supervision. It first identifies SigLIP as a stronger perceptual backbone within the evaluated training setup, then introduces GESSO, a learned perceptual distance that distills teacher-provided score differences into patch-level feature distances. The resulting loss supplements flow-matching and existing perceptual objectives. Experiments across three model scales report improvements in preference metrics and image fidelity, with little change in prompt alignment.

Critical strengths
  • GESSO only brings slight additional integration overhead

GESSO incorporates preference information into generator training by reusing existing SigLIP features, introducing little additional online training cost and no inference overhead.

  • Extensive evaluation and ablation studies.

The authors provide experiment results that cover multiple model scales, teacher choices, projector controls, and independent training runs. Together with the VLM preference comparisons, these results support the consistency of GESSO’s benefits within the evaluated setting.

Critical weaknesses
  • Modest gains and a missing PRISM-Bench ablation.

In Table 4, GESSO improves H/32 pretraining HPSv2.1 from 23.17 to 23.84 (approximately 2.9%), while post-finetuning GenEval increases only from 0.880 to 0.884. These gains appear incremental. Although PRISM-Bench separately evaluates alignment and aesthetics, Tables 6 and 9 omit the PixelGen-SigLIP baseline, preventing attribution of the reported performance to GESSO. The reviewer requests a matched comparison with and without GESSO, reporting changes in both scores to clarify its practical contribution.

  • Insufficient assessment of remaining generation limitations.

The evaluation focuses on one pixel-space model family, while Table 6 shows that the proposed system remains behind several latent-space systems, including FLUX.1-dev, in alignment and aesthetics. Although differences in resolution and training configurations complicate comparison, this gap warrants discussion. The reviewer also finds the red panda’s anatomy and the gear assembly’s mechanical structure in Figure 1 unconvincing. Matched examples with and without GESSO, together with a systematic failure analysis, would help establish whether higher aesthetic scores translate into more structurally coherent images.

Questions for authors

See the Weaknesses for more details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.