- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors present GMGM (Generative Multi-modal Gradient Matching), a data distillation framework for multi-LLMs that produces a synthetic dataset that's aligned with the downstream model's intended task, thus obtaining superior performance at a fraction of the data usage.
Critical strengths
S1: The objective introduced by the manuscript is well grounded in domain and test-time adaptation literature, and subsequently motivated by some measure of theoretical grounding (Theorem 1) which is further expanded on in the appendix. Furthermore, despite necessitating the usage of a frozen text-image critic model, all three pieces of the objective are relatively cheap to compute compared to the rest of the training procedure, given that they are computed mainly on the forward passes, with the backward pass required for gradient matching being limited to a thin linear probe downstream of the vision encoder.
S2: The results reported by the authors are strong, as they seem to suggest the establishing of a new Pareto frontier on the amount of compute. It is particularly remarkable to note that the framework proposed by the authors is not only limited to dataset compression, indeed it seems that generating synthetic data using GMGM produces higher quality text-image pairs than the raw data available. Since there is no limit on the amount of synthetic data that can be generated (as opposed to the finite quantity of samples inherent to any curated dataset), training can be scaled further without collapse in model quality.
Critical weaknesses
W1: Previous work [1] has established the existence of a modality gap in multi-modal self-supervised representation learning models. This gap consists of the misalignment of identical semantical concepts when embedded through e.g. the text or image modality. In practice, the vector embedding the word "car" and an image of a car might reside in separate regions of the learned latent space. Recent work [2] has shown that the misalignment introduced by this modality gap can be reduced, due to it being largely approximated by an orthogonal transformation (in other words, there seems to be an inherent rotation that differentiates between input modalities, and this rotation can be learned and inverted given a batch of data).
In this work, semantic alignment across modalities is one of the three properties that is sought to be preserved by the proposed GMGM objective. Despite this, the authors fail to address this modality gap in their proposed method (Equation 2), despite correctly identifying its presence in the form of the component of . By performing the GMGM update on a mini-batch of prompt-image pairs, or by generating a synthetic minibatch of pairs for each input, the modality gap could be successfully bridged, potentially resulting in decreases in hallucination rate and bias.
W2: One of the central claims of the paper is the reduction of bias inherent in large-scale T2I datasets. Part of this claim is substantiated by the reduced tendency for Hallucinations (Fig. 2a) and the higher level of semantic grounding (Fig. 2b and 3) induced by GMGM finetuning. In the ethics statement and Appendix, they acknowledge that a biased critic might carry his bias over to the GMGM finetuned model. The impact of the chosen reference data is, however, not discussed. Since distillation is well known to amplify model biases[3], and given the potential impact of this work on the field of Generative Models, care should be taken to reduce this tendency as much as possible.
[1] Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. 2022. Mind the gap: understanding the modality gap in multi-modal contrastive representation learning. In Proceedings of the 36th International Conference on Neural Information Processing Systems (NIPS '22). Curran Associates Inc., Red Hook, NY, USA, Article 1280, 17612–17625.
[2] Antonio D’Orazio, Maria Rosaria Briglia, Donato Crisostomi, Dario Loi, Emanuele Rodolà, and Iacopo Masi. 2026. Implicit Inversion turns CLIP into a Decoder. In The Fourteenth International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=hvukI2ws5O
[3] Jaimeen Ahn, Hwaran Lee, Jinhwa Kim, and Alice Oh. 2022. Why Knowledge Distillation Amplifies Gender Bias and How to Mitigate from the Perspective of DistilBERT. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 266–272, Seattle, Washington. Association for Computational Linguistics.
Questions for authors
Q1: Can the authors provide a small-scale study on the presence of a modality gap in the encoded latents, as well as a short ablation showing the potential impact of modality gap removal on the performance of the method? Given the already impressive results in terms of semantic grounding (Fig. 3), the removal of the modality gap could further tighten the upper bound on training risk by shrinking the component, which is currently irreducible by sole optimization of .
Q2: Can the authors comment on the potential impact of a biased reference dataset? In case of presence of potential bias induced by reference data, can the Ethics Statements be extended to include some comments on the existence of this risk, as well as potential mitigations? Do the authors think that, within the time constraints of the short rebuttal period, it would be reasonable to do a qualitative study on the potential biased introduced by GMGM post-distillation, in a similar fashion to the current manuscript's Figure 3?
5 comments
W1 + Q1: Modality-gap diagnosis and calibration
Motivated by Liang et al. (2022) and the orthogonal alignment formulation of D'Orazio et al. (2026, your reference [2]), we measured the image-text modality gap on held-out pairs and tested whether calibrating the global orthogonal rotation between CLIP image and text embeddings improves GMGM synthesis and downstream fine-tuning.
On 4,000 training-domain image-text pairs (deduplicated at the image level and disjoint in prompt templates from the 1,000 evaluation pairs and all downstream benchmarks), we solved the orthogonal Procrustes problem over unit-normalized embeddings and :
\sum_{i\in\mathcal C_{\mathrm{fit}}} \\|Q\widehat h_i-\widehat u_i\\|_2^2.$$ Freezing $Q^*$ during synthesis, we replaced Eq. 2 with $$L_{\mathrm{align}}^{Q}(z;t) =1-\cos\\!\left(Q^*h(\mathcal G(z;t)),u(t)\right),$$ while holding the generator, SAE inputs, proxy-gradient objective, prompts, initial latents, and reference batches unchanged. ### Held-out geometry | Alignment representation | Mean real-pair cosine distance | Image/text mean separation | Image-to-text / text-to-image retrieval | |---|---:|---:|---:| | Original CLIP alignment | 0.682 | 0.61 | 0.21 / 0.19 | | Orthogonally calibrated alignment | 0.541 | 0.28 | 0.29 / 0.27 | Evaluated on the same 1,000 disjoint held-out pairs (where mean separation is the cosine distance between modality centroids and retrieval is bidirectional Recall@1), orthogonal calibration cuts centroid separation in more than half (0.61 to 0.28) and lifts Recall@1 by +8 percentage points in both directions. ### Matched synthesis and fine-tuning We then modified only Eq. 2 in a matched 5K synthesis and fine-tuning ablation (SDXL-1.0, $K=50$ latent steps), evaluating image-instruction consistency with the external Qwen2.5-VL-72B judge: | GMGM semantic alignment | Winoground | POPE F1 | MMBench | Independent image-instruction consistency | |---|---:|---:|---:|---:| | Original | 12.8 | 85.1 | 62.8 | 0.82 | | Calibrated | 13.4 (+0.6) | 85.6 (+0.5) | 63.4 (+0.6) | 0.85 (+0.03) | Shrinking the modality gap yields consistent gains across all three downstream benchmarks (+0.6 / +0.5 / +0.6 points) and raises external image-instruction consistency by +3 percentage points. In the theoretical analysis, this calibration acts directly on the real-pair alignment residual $\delta_{\mathrm{real}}=\mathbb E_{(x_r,t)\sim P_{\mathrm{real}}}[1-\cos(h(x_r),u(t))]$ in Lemma 1. Optimizing synthetic latents $z$ lowers $L_{\mathrm{align}}$ while leaving real embeddings untouched, whereas fitting $Q^*$ reduces $\delta_{\mathrm{real}}$ on held-out pairs by 0.141 (from 0.682 to 0.541). In our submitted proof (Theorem 2, Step 1), $\delta_{\mathrm{real}}$ was absorbed into the optimal joint task error $\lambda_{\mathrm{opt}}^*$. `[MAIN-P5]`, `[APP-P14]`, `[APP-P15]`, `[APP-P24]`, and `[APP-P25]` Revised Lemma 1 (Appendix B.2) retains $\delta_{\mathrm{real}}$ and its calibrated counterpart $\delta_{\mathrm{real}}^Q$ explicitly in the paired embedding bound $\mathbb E\\|h(x_s)-h(x_r)\\|_2 \le \sqrt{2\mathbb E[L_{\mathrm{align}}]} + \sqrt{2\delta_{\mathrm{real}}}$, keeping this modality-gap residual separate from $\lambda_{\mathrm{opt}}^*$, while Sec. 3.2 and Appendix J report the calibration ablation. For the complementary RKHS decomposition of our fixed-reference SAE objective, please also see our response to **Reviewer Hwax, W1 + Q1: What the singleton objective controls**.W2 + Q2: Reference-induced scene bias and mitigation
Although reference images in are never used directly in MLLM fine-tuning, they shape synthesized training data through both the SAE feature target (Eq. 3) and the mean proxy-gradient target (Eq. 5). Skew in the reference pool can therefore propagate into generated images and persist after fine-tuning, analogous to bias amplification in distillation (Ahn et al., 2022, your reference [3]).
GMGM draws 50 reference images per super-category from the ImageNet-21k training split, whose annotations provide object synsets without scene labels. To run a controlled intervention within the rebuttal window that quantifies bias propagation, we labeled candidate reference images and all generated images using a Places365 classifier (Zhou et al., 2018) with its official indoor/outdoor scene hierarchy and top-10 voting rule. Within a super-category group spanning both contexts, we drew 50 references per condition from a shared pool (an indoor-skewed 45:5 split vs. a balanced 25:25 split) alongside passive generation, holding prompts, initial latents, synthesis settings, and the LoRA recipe fixed. We audited generated scene shares across 200 images (100 probe prompts 2 latent seeds) and evaluated post-fine-tuning indoor/outdoor scene-classification accuracies on a balanced held-out test set:
| Reference condition (indoor:outdoor) | Generated indoor share | Indoor / outdoor accuracy | Group gap |
|---|---|---|---|
| Indoor-skewed (45:5) | 86% | 73% / 56% | 17 pp |
| Rebalanced (25:25) | 49% | 70% / 64% | 6 pp |
| Passive generation (no references) | 47% | 69% / 58% | 11 pp |
Under a 90% indoor reference batch (45:5), prompts without explicit scene constraints default heavily to indoor backgrounds (86% indoor share vs. 47% under passive generation), creating a 17 percentage-point downstream accuracy gap (73% indoor vs. 56% outdoor). Stratifying the reference batch to 25:25 restores balanced generation (49% indoor) and narrows the downstream subgroup gap from 17 to 6 percentage points, lifting outdoor accuracy from 56% to 64% while indoor accuracy shifts from 73% to 70%. [APP-P25] Appendix K details this controlled intervention and its labeling and evaluation protocol.
[MAIN-P11] We have also expanded the Ethics Statement with the following paragraph:
Skew in the real reference batch can also propagate through the SAE feature and mean proxy-gradient anchors (Eqs. 3 and 5), biasing synthesized scene distributions and downstream error rates even though reference images are never used directly in fine-tuning. In a controlled indoor-outdoor case study (Appendix K), rebalancing a skewed 45:5 reference split to 25:25 narrows the post-fine-tuning group-accuracy gap from 17 to 6 percentage points. When reliable attribute labels and sufficient images from each subgroup exist within a category, we recommend stratified reference sampling alongside subgroup accuracy reporting; when subgroup representation in the pool is sparse, practitioners should expand reference coverage before synthesis.
References:
- Liang et al. (2022). Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. NeurIPS.
- D'Orazio et al. (2026). Implicit Inversion Turns CLIP into a Decoder. ICLR.
- Ahn et al. (2022). Why Knowledge Distillation Amplifies Gender Bias and How to Mitigate from the Perspective of DistilBERT. GeBNLP Workshop, ACL, pp. 266-272.
- Zhou et al. (2018). Places: A 10 Million Image Database for Scene Recognition. IEEE TPAMI, 40(6), 1452-1464.
We appreciate your constructive feedback and the insightful connections to prior work on the CLIP modality gap and bias amplification in distillation. All corresponding updates have been incorporated into the revised manuscript and appendix (highlighted in red in the updated PDF); throughout this response, tags such as
[MAIN-P5]and[APP-P24]denote continuous PDF page numbers in the main paper and appendix, respectively.