- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on reweighting the synthetic data when mixing it with the original training set for downstream training. The proposed Informative Data Reweighting (IDR) is based on the Information Bottleneck (IB) principle. Finally, the experiments show some improvement across different backbones.
Critical strengths
- This paper incorporates the information bottleneck and provides extensive theoretical justification in the main paper and appendix, making the paper supported by a systems-theory scope.
- This paper, regarding the theory side, provides very comprehensive background. As an image classification method, it even provides background knowledge (including the training procedure) on diffusion models in the appendix.
Critical weaknesses
- This paper uses almost one page to claim its contribution in the introduction section, making the presentation long-winded and lacking focus. The contribution should be highlighted and summarized in a clear and accurate manner.
- This paper shows severe experimental problems, as it claims to be a data-augmentation-based reweighting method and provides extensive mathematical derivation. The experiments use very old and limited baselines and ignore most of the new data augmentation methods, especially as the more and more excellent generative models today, making this paper's experimental evidence out of date.
- Extensive theoretical analysis coupled with very weak experimental support diminishes the credibility of the claim.
Questions for authors
Given that the analysis of the information bottleneck is so comprehensive, why hasn't there been consideration of applying it to broader scenarios—such as the current practice of incorporating synthetic data into the training of LLMs and generative models?
5 comments
Response to Review by Reviewer ojjJ - Part 2
W3. Strength of the experimental support.
The experimental support is broad and statistically tested. Reviewers w6zi and pw8Y agree, writing "The experiments are sufficient and the results are competitive" and "The empirical evaluation is broad", respectively. On ImageNet-1K, Table 1 in Section 4.1 of the paper shows that IDR improves top-1 by 1.1 percentage points over the best baseline on each of ViT-S/B and Swin-T/B. Over ten seeds, Table 11 in Section D.8 of the paper gives .
Matched budgets. The full sweep in Table 16 in Section D.16 shows that IDR leads CBF and GenDataAgent at every budget from to . We have added all competing methods at a common budget to Table 26 in Section D.24 of the appendix of the revised paper, and Section 4.2 summarizes this comparison. IDR reaches 85.3 vs. 84.5 for GenDataAgent, with 2.0% more selected-data generation-plus-fine-tuning compute. The cost breakdown is in Section D.17, and these costs exclude shared LDM training and budget search.
Generators and breadth. The evaluation covers DiT-B/L/XL, DDPM and StyleGAN-XL in Tables 8 and 13 of the paper, Swin-T trained from scratch, COCO, ADE20K, Flowers/Pet/Cars, ImageNet-A/R, two medical datasets, and augmentation and active-learning comparisons. IDR also complements recent augmentation. In Table 4 in Section D.1 of the paper, IDR+PAA reaches 86.1, vs. 85.6 for IDR and 84.5 for PAA.
New experiments directly support the claims. In response to Reviewers w6zi and pw8Y, we added the revised feature-dependent VUIB to Sections 3.2–3.3 and its proofs and scope to Appendix E. Table 25 in Section D.23 of the appendix reports 85.6% at , 1.1 percentage points above GenDataAgent, and these revised-objective results are distinguished from the submitted IDR results. The generator and centroid experiments have been added to Tables 27 and 20 in Sections D.25 and D.19 of the appendix of the revised paper, respectively, and the beginning of Section 4 lists them. The summary below shows IDR staying ahead with DiT-XL, improving DA-Fusion and exceeding centroid alternatives. The code is in the anonymized repository linked in the paper.
| New experiment (ViT-B) | Best competing method | IDR (Ours) |
|---|---|---|
| Revised VUIB, | GenDataAgent: 84.5 | 85.6 |
| DiT-XL synthetic data | GenDataAgent: 84.8 | 85.7 |
| Stable Diffusion images (DA-Fusion), | DA-Fusion: 84.1 | 84.8 |
| Class-centroid regularization, | Centroid-similarity weights: 84.3 | 85.3 |
Q1. Broader scenarios: LLMs and generative models.
We focused on IDR's setting: training predictive models on real plus generated images. The scenarios you mention involve training LLMs and generative models on synthetic data, a different problem from deciding how much each sample should count for a predictive model. The two are complementary. Any better generator can supply IDR's data, and Tables 8 and 13 of the paper show that IDR is generator-agnostic across DiT-B/L/XL, DDPM and StyleGAN-XL.
Both scenarios are promising future work. They require a suitable label-like relevance variable and a softly quantizable representation. For LLM fine-tuning on synthetic instruction or reasoning data, could be the task category or verifier outcome, with prototypes over sequence embeddings and learned weights scaling the per-sample loss. For generative-model training, could be the conditioning variable, with prototypes over denoiser features and learned weights scaling the denoising loss. This scope and these possible extensions have been added to Section G.2 of the appendix of the revised paper, and we make no empirical claim for these untested settings.
References
[1] He et al. Is synthetic data from generative models ready for image recognition? ICLR 2023
[2] Chhabra et al. "What data benefits my classifier?" Enhancing model performance and interpretability through influence-based data selection. ICLR 2024
[3] Jain et al. Learning model uncertainty as variance-minimizing instance weights. ICLR 2024
[4] Wang et al. Informative synthetic data generation for thorax disease classification. UAI 2025
[5] Li et al. GenDataAgent: On-the-fly dataset augmentation with synthetic data. ICLR 2025
[6] Shu et al. Meta-Weight-Net: Learning an explicit mapping for sample weighting. NeurIPS 2019
[7] Guo et al. Learning to re-weight examples with optimal transport for imbalanced classification. NeurIPS 2022
[8] Kim et al. Configuring data augmentations to reduce variance shift in positional embedding of vision transformers. AAAI 2025
[9] Kang et al. Model-agnostic and efficient mixup augmentation guided by saliency maps. IEEE TPAMI 2025
[10] Gwon et al. Enhancing cost efficiency in active learning with candidate set query. TMLR 2025
[11] Peebles and Xie. Scalable diffusion models with transformers. ICCV 2023
[12] Ho et al. Denoising diffusion probabilistic models. NeurIPS 2020
[13] Sauer et al. StyleGAN-XL: Scaling StyleGAN to large diverse datasets. SIGGRAPH 2022
[14] Trabucco et al. Effective data augmentation with diffusion models. ICLR 2024
[15] Rombach et al. High-resolution image synthesis with latent diffusion models. CVPR 2022
Response to Review by Reviewer ojjJ - Part 1
Thank you for the review and for recognizing our information-bottleneck foundation and thorough background. IDR trains a predictive model on real plus generated images and learns how much each sample counts.
W1. The contribution statement is too long.
We have revised the contribution statement in Section 1 of the revised paper. We removed a sentence that repeated the results claim of the abstract, and the second contribution now states the revised VUIB, whose compression term is computed exactly and depends on the learned features, together with its corrected computational complexity. The key findings on the learned weights and on matched budgets and compute are now summarized in Sections 4.1 and 4.2, and the list at the beginning of Section 4 points to every added experiment in Sections D.18–D.25 of the appendix.
W2. Baselines and generative models.
IDR's baselines are recent. Five of the seven data selection/reweighting baselines and both classifier-augmentation baselines date from 2023 to 2025.
The advances you mention concern training generative models, while IDR trains a predictive model. Stronger generators and synthetic-data training for LLMs and generative models either improve a generator or train one. IDR solves a different problem: it learns how much each real or generated sample counts when training an image classifier. The two are complementary, not competing. Since IDR is generator-agnostic, any better generator can supply its data.
IDR is compared against its direct competitors, methods that select, reweight or augment a classifier's training data. Table 1 in Section 4.1 of the paper covers data selection and reweighting with CBF [1] (ICLR 2023), IE [2] and REVAR [3] (ICLR 2024), IDS [4] (UAI 2025) and GenDataAgent [5] (ICLR 2025), plus MW-Net [6] (NeurIPS 2019) and OTR [7] (NeurIPS 2022) as established references. For classifier augmentation, Table 4 in Section D.1 of the paper shows PAA [8] (AAAI 2025) at 84.5 and GuidedMixup [9] (TPAMI 2025) at 84.3, against 85.6 for IDR. Combined with IDR, they reach 86.1 and 86.0. The active-learning baselines in Table 10 in Section D.7 of the paper, including CSQ [10] (TMLR 2025), reach at most 84.4.
IDR is best with every generator. Table 16 in Section D.16 of the paper shows that with DiT-B [11] on ViT-B, IDR leads GenDataAgent by 0.8 percentage points at a matched 3N budget. With DDPM [12] and StyleGAN-XL [13] on Swin-B, Table 13 in Section D.12 of the paper puts IDR ahead of the strongest baseline by 1.1 and 1.2 points, respectively. Table 8 in Section D.5 of the paper shows IDR reaching 85.7 with DiT-L/XL.
Stronger generators and a recent generative augmentation method. We have conducted two additional experiments with ViT-B. We trained CBF and GenDataAgent on DiT-XL data at their cross-validated budgets. We also compared with DA-Fusion [14] (ICLR 2024), which edits real images with Stable Diffusion [15], and used IDR to reweight its images at a fixed 1N budget. IDR stays best with DiT-XL, 0.9 percentage points above the best baseline, and improves DA-Fusion by 0.7 points. These results have been added to Table 27 in Section D.25 of the appendix of the revised paper. The DiT-XL budgets differ across methods as shown below, and Table 26 in Section D.24 gives the comparison under a common budget.