acceptodds
Withdrawn from ICLR 2027

TASTE: Training-Aware Data Selection for Text-to-Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The remarkable progress of text-to-image (T2I) generative models, such as Imagen, Stable Diffusion, and FLUX, has led to substantial improvements in visual quality. However, their performance is fundamentally limited by the quality of training data. Web-crawled and synthetic image datasets often contain low-quality or redundant samples which lead to degraded visual fidelity, unstable training, and inefficient computation. Hence, effective data selection is crucial for improving data efficiency. Existing approaches rely on costly manual curation or heuristic scoring based on single-dimensional features in text-to-image data filtering. Although meta-learning-based data selection has been explored for LLMs, its application to text-to-image model training remains underexplored. To this end, we propose **TASTE**, a meta-gradient-based framework to select a suitable subset from large-scale text-image data pairs. Our approach automatically estimates the influence of each sample by iteratively optimizing the model from a data-centric perspective. TASTE consists of two key stages: data rating and data pruning. We train a lightweight rater to estimate each sample's influence based on gradient information, enhanced with multi-granularity perception. We then use the Shift-Gsample strategy to select informative subsets. Experiments on both synthetic and web-crawled datasets demonstrate that Taste consistently improves downstream model performance. Training on the 50% subset selected by TASTE achieves performance comparable to full-data training while improving data efficiency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image generation, Data selection; Coreset selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rPRAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces TASTE, a data selection method for T2I model pretraining. Instead of just use heuristic scores such as aesthetic score or clarity score, it rates each sample by how a proxy T2I model reacts to them during training. A light rater is trained with a first order metalearning signal that comes from 2 copies of the proxy model, one of which is also guided by a validation set. Those top-rated samples turn out to be plain, easy images which hurt training, so the final subset is drawn with a new sampling rule called Shift-GSample that skips the head of the ranking and samples around the middle (slightly shift upward) part. The athors show that a half-size TASTE subset beats random selection and is comparable to full-data training, across several model sizes, datasets and retention ratios.

In conclusion, this paper shows:

  1. A data rater can be learned from a proxy T2I model's training behaviour with a cheap first-order update.
  2. Top-Kon this learned score fails, because the top samples are plain and easy, while sampling the middle part of the ranking works.
  3. The selected subset beats random selection, and the half-size subset is comparble to full data with half of the T2I training time.
Critical strengths
  • The question is real and well-framed. Heuristic scores such as aesthetic or clarity score are not optimized for the downstream training, and " let the T2I model itself tell which sample are helpful" is a legit and more general direction than any fixed scorer learned from external annotations. This is the main reason I think this problem itself is worth working on even if the current method does not really support it.
  • The improvement over random selection is consistent asTASTE wins in ever paired comparison agains random selection. This is the strongest evidence in this paper.
  • The gsample/shift-gsample idea for handling a score that "most top rated actually means bad in another way" actually a very good example for other scorer works or data selection works. If your score is something the model may be hacking/cheating, directly
Critical weaknesses
  1. The learned rater doesn't give a reasonable ranking and Top-K in such rater just not work. The paper claim the high-score samples are plain/simple images that are easy to fit (3.2, Fig 4), and an additional pruning strategy is required (App. B.2). This basically means the whole TASTE training pipeline does not give out a reasonable score, and the real contribution becomes the sampling rule after it. Unfortunately, the math idea or any theory analysis (or even hypotehsis) are not well framed in the paper
  2. The validation set for TASTE training is only a random hold-out from the noisy source, this means the meta objective may naturally prefer easy, low-loss images which cause the "rater prefers plain images" phenomenon. Intuitively the val set should be some known-good or target good data, but the paper never discusses any stuff about val set selection, only size got varied.
  3. The shift-Gsample appears from nowhere and this paper provide NO analysis, this raise some concern about some important engineering deicision didn't get properly justified in a research target, math analysis or proper justification is needed, "it is just better" can't be a good reason. And even with all the components together TASTE still not beat the Aesthetic score + plain Top-K at the 6M budget. TASTE only wins at 15M. ALSO, the result of aesthetic score + plain top-k with 6M budget is only in appendix Table 8, the main table shows "Ours-small 6M" without this stronger comparator which can invalidate the claim, statement and result in the main text. Also flag a possible severe misconduction.
  4. The direction about "use the model's own training behaviour to judge which samples are good" is not new [2-10] but none of these works is compared in results or method design, and learned (or meta-gradient) selectors are even excluded from the baselines (App. A.4). For example: [2, 3] are almost the same rater + validation gradient design, and [8, 9] already explain why the easiest samples are not the most useful. On the fixed scorer side: only one set of learned scorer (LAION aesthetic) is tested while we haev tons of aesthetic/IQA/Preference scorers [11-14] and CLIP-score filtering. As the sampler also helps hand-crafted baselines (App. Table 8) so with more scoreres + the same sampler the gap in point 3 may get worse
  5. The training cost of TASTE itself for data selection is not discussed properly, considering the dataset transferatino (same scorer but for different set) is not properly discussed, assume such data selection pipeline need to be trained and run for every new image set, the cost may need to be counted as part of final training.
  6. The evaluation all comes from known problematic eval protocol such as FID, CLIP-score or GenEval which are known to be fragile and not really reflecting the quality of final result. [15-23] And the gain is mostly FID while CLIP and GenEVAL stay below full data. Also the sampler is tuned on MJHQ which is the test set, which means TASTE may just select samples close to the test distribution, while matching the distribution alone can already improve FID as that's how it been defined and there is some prompt-side distribution alignment research have shown [1]. Also, as the images are all recaptioned by CogVLM, it may just select images that CogVLM captions better, not really "better images for training T2I".

There are also tons of other tiny weakness like the pretrained T2I are all AR model and no standard Diffusion T2I or so, but this kind of weakness not really directly affect the claim.

Ref:

  • [1] SY. Yeh et al. TIPO: Text to Image with Text Presampling for Prompt Optimization. ICLR 2026
  • [2] M. Ren et al. Learning to Reweight Examples for Robust Deep Learning. ICML 2018
  • [3] J. Shu et al. Meta-Weight-Net: Learning an Explicit Mapping for Sample Weighting. NeurIPS 2019
  • [4] P. W. Koh and P. Liang. Understanding Black-box Predictions via Influence Functions. ICML 2017
  • [5] A. Ghorbani and J. Zou. Data Shapley: Equitable Valuation of Data for Machine Learning. ICML 2019
  • [6] M. Toneva et al. An Empirical Study of Example Forgetting during Deep Neural Network Learning. ICLR 2019
  • [7] S. Swayamdipta et al. Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. EMNLP 2020
  • [8] B. Sorscher et al. Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning. NeurIPS 2022
  • [9] S. Mindermann et al. Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt. ICML 2022
  • [10] T. Evans et al. Data Curation via Joint Example Selection Further Accelerates Multimodal Learning. NeurIPS 2024
  • [11] J. Ke et al. MUSIQ: Multi-scale Image Quality Transformer. ICCV 2021
  • [12] H. Wu et al. Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels. ICML 2024
  • [13] J. Xu et al. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. NeurIPS 2023
  • [14] Y. Kirstain et al. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. NeurIPS 2023
  • [15] G. Stein et al. Exposing Flaws of Generative Model Evaluation Metrics and Their Unfair Treatment of Diffusion Models. NeurIPS 2023
  • [16] T. Kynkäänniemi et al. The Role of ImageNet Classes in Fréchet Inception Distance. ICLR 2023
  • [17] S. Jayasumana et al. Rethinking FID: Towards a Better Evaluation Metric for Image Generation. CVPR 2024
  • [18] G. Parmar et al. On Aliased Resizing and Surprising Subtleties in GAN Evaluation. CVPR 2022
  • [19] Z. Lin et al. Evaluating Text-to-Visual Generation with Image-to-Text Generation. ECCV 2024
  • [20] M. Yuksekgonul et al. When and Why Vision-Language Models Behave like Bags-of-Words, and What to Do About It? ICLR 2023
  • [21] M. Otani et al. Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation. CVPR 2023
  • [22] Y. Hu et al. TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering. ICCV 2023
  • [23] GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation. arXiv:2512.16853, 2025
Questions for authors
  1. Please position TASTE against meta-learned re-weighting [2, 3] and reference-model based selection [9, 10], and compare against several existing scorers [11–14] with the same sampler at several budgets.
  2. If the validation set is a curated or target-domain set instead of a random hold-out, does the rater still prefer plain images?
  3. Which real image set is the FID reference, and can the authors add one more metric such as FDD, KDD or a small human study? Does the rater see the caption, since section 3.1 says image-text pairs while Alg. 1 and Fig. 2a show image only input?
  4. What is the full end-to-end cost in GPU hours, including the proxy warm-up and the downstream runs used to tune the sampler?

Besides the questions above, please also refer to the weakness part.

Ref (same numbering as in Weaknesses)

  • [2] M. Ren et al. Learning to Reweight Examples for Robust Deep Learning. ICML 2018
  • [3] J. Shu et al. Meta-Weight-Net: Learning an Explicit Mapping for Sample Weighting. NeurIPS 2019
  • [9] S. Mindermann et al. Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt. ICML 2022
  • [10] T. Evans et al. Data Curation via Joint Example Selection Further Accelerates Multimodal Learning. NeurIPS 2024
  • [11] J. Ke et al. MUSIQ: Multi-scale Image Quality Transformer. ICCV 2021
  • [12] H. Wu et al. Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels. ICML 2024
  • [13] J. Xu et al. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. NeurIPS 2023
  • [14] Y. Kirstain et al. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. NeurIPS 2023
Reviewer_knDoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TASTE, a meta-gradient-based framework for selecting useful subsets from large-scale image–text datasets. The framework consists of two stages: data rating, which learns sample scores using validation-guided feedback from a proxy model, and data pruning, which constructs a subset from the learned ranking. Experiments show that training on a TASTE-selected subset containing 50% of the data achieves broadly comparable performance to full-data training, with improved FID and reduced downstream training time.

Critical strengths
  • Selecting valuable training data to avoid wasting compute is useful, especially when training large models. Although it remains unclear whether this approach can be applied to larger T2I models, I acknowledge that the authors leave this investigation for future work.
  • The paper is easy to follow.
Critical weaknesses

The benefits appear to vary across evaluation metrics. In Table 1, TASTE improves FID and CLIP over random selection, but its GenEval score is nearly identical to the random baseline (0.2645 vs. 0.2632). These results do not demonstrate a clear improvement in the compositional aspects of prompt following measured by GenEval. This matters because prompt following is important for T2I performance.

Questions for authors

Could the authors discuss why the improvements over random selection are clearer on FID and CLIP than on GenEval?

Reviewer_K5Z4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TASTE, a meta-gradient framework that trains a lightweight rater to score each text-image pair by how much it helps a proxy model’s validation loss, then selects a subset with Shift-Gsample, which discards the top 25% (plain, redundant images) and samples the rest around the middle-to-late ranks. On LAION-30M, a 15M subset improves FID over full-data training with CLIP and GenEval slightly lower, beats random and heuristic baselines at the same budget, and transfers across STAR model scales, FLUX-mini, and other datasets.

Critical strengths

The method targets a real and valuable problem, and its design captures the characteristics of the image modality. Most T2I data filtering relies on fixed, single-dimension scores such as aesthetics or clarity, which have no direct relationship to downstream training performance. TASTE instead uses “how much a sample helps reduce validation loss” as its scoring objective. More importantly, the authors identify and validate a phenomenon that is meaningful for images: the highest-scoring samples are often merely visually simple and easy to fit, rather than informative.

Critical weaknesses

The paper says that half the data matches full-data training, but the evidence is not solid enough. The key hyperparameters were tuned on FID of the same benchmark used to report the main results, so model selection and evaluation overlap. All results are single runs without seeds or error bars, so it is unclear whether the small gap to the aesthetic baseline exceeds noise. The subset is also slightly worse than full training on CLIP and GenEval and better only on FID, yet the abstract invites a stronger reading. Besides, the proxy and downstream models belong to the same family and are validated only up to 0.9B, leaving larger models untested, even though data selection matters most at large scale.

Questions for authors
  1. The hyperparameters (cutoff ratio, sampling center, spread) were selected using FID on MJHQ-30K, which is also the benchmark for the main results. Could you re-select them on a held-out development set and report the results?
  2. Are all results from a single run? Could you report means and standard deviations over multiple seeds, especially for the comparison with the aesthetic baseline and the GenEval gap?
  3. Figure 3 shows that high-scored samples are mostly low-loss ones. How strongly does the learned score correlate with per-sample loss, and how does ranking directly by loss combined with the same pruning strategy perform?
  4. Does the gain come from the rater or from the pruning strategy? If heuristic rankings such as aesthetic or frequency are combined with the same pruning strategy, how large is the gap to TASTE’s ranking?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.