- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces TASTE, a data selection method for T2I model pretraining. Instead of just use heuristic scores such as aesthetic score or clarity score, it rates each sample by how a proxy T2I model reacts to them during training. A light rater is trained with a first order metalearning signal that comes from 2 copies of the proxy model, one of which is also guided by a validation set. Those top-rated samples turn out to be plain, easy images which hurt training, so the final subset is drawn with a new sampling rule called Shift-GSample that skips the head of the ranking and samples around the middle (slightly shift upward) part. The athors show that a half-size TASTE subset beats random selection and is comparable to full-data training, across several model sizes, datasets and retention ratios.
In conclusion, this paper shows:
- A data rater can be learned from a proxy T2I model's training behaviour with a cheap first-order update.
- Top-Kon this learned score fails, because the top samples are plain and easy, while sampling the middle part of the ranking works.
- The selected subset beats random selection, and the half-size subset is comparble to full data with half of the T2I training time.
Critical strengths
- The question is real and well-framed. Heuristic scores such as aesthetic or clarity score are not optimized for the downstream training, and " let the T2I model itself tell which sample are helpful" is a legit and more general direction than any fixed scorer learned from external annotations. This is the main reason I think this problem itself is worth working on even if the current method does not really support it.
- The improvement over random selection is consistent asTASTE wins in ever paired comparison agains random selection. This is the strongest evidence in this paper.
- The gsample/shift-gsample idea for handling a score that "most top rated actually means bad in another way" actually a very good example for other scorer works or data selection works. If your score is something the model may be hacking/cheating, directly
Critical weaknesses
- The learned rater doesn't give a reasonable ranking and Top-K in such rater just not work. The paper claim the high-score samples are plain/simple images that are easy to fit (3.2, Fig 4), and an additional pruning strategy is required (App. B.2). This basically means the whole TASTE training pipeline does not give out a reasonable score, and the real contribution becomes the sampling rule after it. Unfortunately, the math idea or any theory analysis (or even hypotehsis) are not well framed in the paper
- The validation set for TASTE training is only a random hold-out from the noisy source, this means the meta objective may naturally prefer easy, low-loss images which cause the "rater prefers plain images" phenomenon. Intuitively the val set should be some known-good or target good data, but the paper never discusses any stuff about val set selection, only size got varied.
- The shift-Gsample appears from nowhere and this paper provide NO analysis, this raise some concern about some important engineering deicision didn't get properly justified in a research target, math analysis or proper justification is needed, "it is just better" can't be a good reason. And even with all the components together TASTE still not beat the Aesthetic score + plain Top-K at the 6M budget. TASTE only wins at 15M. ALSO, the result of aesthetic score + plain top-k with 6M budget is only in appendix Table 8, the main table shows "Ours-small 6M" without this stronger comparator which can invalidate the claim, statement and result in the main text. Also flag a possible severe misconduction.
- The direction about "use the model's own training behaviour to judge which samples are good" is not new [2-10] but none of these works is compared in results or method design, and learned (or meta-gradient) selectors are even excluded from the baselines (App. A.4). For example: [2, 3] are almost the same rater + validation gradient design, and [8, 9] already explain why the easiest samples are not the most useful. On the fixed scorer side: only one set of learned scorer (LAION aesthetic) is tested while we haev tons of aesthetic/IQA/Preference scorers [11-14] and CLIP-score filtering. As the sampler also helps hand-crafted baselines (App. Table 8) so with more scoreres + the same sampler the gap in point 3 may get worse
- The training cost of TASTE itself for data selection is not discussed properly, considering the dataset transferatino (same scorer but for different set) is not properly discussed, assume such data selection pipeline need to be trained and run for every new image set, the cost may need to be counted as part of final training.
- The evaluation all comes from known problematic eval protocol such as FID, CLIP-score or GenEval which are known to be fragile and not really reflecting the quality of final result. [15-23] And the gain is mostly FID while CLIP and GenEVAL stay below full data. Also the sampler is tuned on MJHQ which is the test set, which means TASTE may just select samples close to the test distribution, while matching the distribution alone can already improve FID as that's how it been defined and there is some prompt-side distribution alignment research have shown [1]. Also, as the images are all recaptioned by CogVLM, it may just select images that CogVLM captions better, not really "better images for training T2I".
There are also tons of other tiny weakness like the pretrained T2I are all AR model and no standard Diffusion T2I or so, but this kind of weakness not really directly affect the claim.
Ref:
- [1] SY. Yeh et al. TIPO: Text to Image with Text Presampling for Prompt Optimization. ICLR 2026
- [2] M. Ren et al. Learning to Reweight Examples for Robust Deep Learning. ICML 2018
- [3] J. Shu et al. Meta-Weight-Net: Learning an Explicit Mapping for Sample Weighting. NeurIPS 2019
- [4] P. W. Koh and P. Liang. Understanding Black-box Predictions via Influence Functions. ICML 2017
- [5] A. Ghorbani and J. Zou. Data Shapley: Equitable Valuation of Data for Machine Learning. ICML 2019
- [6] M. Toneva et al. An Empirical Study of Example Forgetting during Deep Neural Network Learning. ICLR 2019
- [7] S. Swayamdipta et al. Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. EMNLP 2020
- [8] B. Sorscher et al. Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning. NeurIPS 2022
- [9] S. Mindermann et al. Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt. ICML 2022
- [10] T. Evans et al. Data Curation via Joint Example Selection Further Accelerates Multimodal Learning. NeurIPS 2024
- [11] J. Ke et al. MUSIQ: Multi-scale Image Quality Transformer. ICCV 2021
- [12] H. Wu et al. Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels. ICML 2024
- [13] J. Xu et al. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. NeurIPS 2023
- [14] Y. Kirstain et al. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. NeurIPS 2023
- [15] G. Stein et al. Exposing Flaws of Generative Model Evaluation Metrics and Their Unfair Treatment of Diffusion Models. NeurIPS 2023
- [16] T. Kynkäänniemi et al. The Role of ImageNet Classes in Fréchet Inception Distance. ICLR 2023
- [17] S. Jayasumana et al. Rethinking FID: Towards a Better Evaluation Metric for Image Generation. CVPR 2024
- [18] G. Parmar et al. On Aliased Resizing and Surprising Subtleties in GAN Evaluation. CVPR 2022
- [19] Z. Lin et al. Evaluating Text-to-Visual Generation with Image-to-Text Generation. ECCV 2024
- [20] M. Yuksekgonul et al. When and Why Vision-Language Models Behave like Bags-of-Words, and What to Do About It? ICLR 2023
- [21] M. Otani et al. Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation. CVPR 2023
- [22] Y. Hu et al. TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering. ICCV 2023
- [23] GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation. arXiv:2512.16853, 2025
Questions for authors
- Please position TASTE against meta-learned re-weighting [2, 3] and reference-model based selection [9, 10], and compare against several existing scorers [11–14] with the same sampler at several budgets.
- If the validation set is a curated or target-domain set instead of a random hold-out, does the rater still prefer plain images?
- Which real image set is the FID reference, and can the authors add one more metric such as FDD, KDD or a small human study? Does the rater see the caption, since section 3.1 says image-text pairs while Alg. 1 and Fig. 2a show image only input?
- What is the full end-to-end cost in GPU hours, including the proxy warm-up and the downstream runs used to tune the sampler?
Besides the questions above, please also refer to the weakness part.
Ref (same numbering as in Weaknesses)
- [2] M. Ren et al. Learning to Reweight Examples for Robust Deep Learning. ICML 2018
- [3] J. Shu et al. Meta-Weight-Net: Learning an Explicit Mapping for Sample Weighting. NeurIPS 2019
- [9] S. Mindermann et al. Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt. ICML 2022
- [10] T. Evans et al. Data Curation via Joint Example Selection Further Accelerates Multimodal Learning. NeurIPS 2024
- [11] J. Ke et al. MUSIQ: Multi-scale Image Quality Transformer. ICCV 2021
- [12] H. Wu et al. Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels. ICML 2024
- [13] J. Xu et al. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. NeurIPS 2023
- [14] Y. Kirstain et al. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. NeurIPS 2023