acceptodds
Under review as a conference paper at ICLR 2027

Generative Multi-modal Gradient Matching

Abstract

Multi-modal Large Language Models (MLLMs) rely on massive web-scraped corpora that are noisy, redundant, and weakly aligned with downstream objectives, leading to persistent failure modes such as object hallucination and brittle compositional generalization. Conventional text-to-image (T2I) synthesis offers a scalable alternative but remains passive: prompts drive sampling without feedback on training utility, producing images that are visually plausible yet redundant for fine-tuning. We present Generative Multi-modal Gradient Matching (GMGM), an active synthesis framework that, given a frozen T2I generator and a small real reference batch, optimizes per-prompt latent codes so that each generated image is simultaneously: (i) prompt-faithful, (ii) feature-aligned with real data via Sparse Autoencoder (SAE) distribution matching, and (iii) gradient-informative for downstream MLLM fine-tuning. Our key insight is that synthetic data should be optimized not merely for perceptual realism, but for alignment with the downstream model's learning dynamics. We prove that minimizing this tri-objective tightens the upper bound on the generalization gap, isolating covariate shift, feature coverage, and trajectory divergence. Empirically, relative to a strong prompt-matched synthetic baseline (TreeSynth), GMGM improves hallucination robustness (+5.6% POPE F1) and compositional reasoning (+58.1% Winoground), while matching its full-budget performance using only 10% of the synthetic data budget. Further experiments demonstrate GMGM's strong cross-architecture generalization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
synthetic data generation, multimodal large language models, data-efficient fine-tuning, gradient matching, dataset distillation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_pWEdOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors present GMGM (Generative Multi-modal Gradient Matching), a data distillation framework for multi-LLMs that produces a synthetic dataset that's aligned with the downstream model's intended task, thus obtaining superior performance at a fraction of the data usage.

Critical strengths

S1: The objective introduced by the manuscript is well grounded in domain and test-time adaptation literature, and subsequently motivated by some measure of theoretical grounding (Theorem 1) which is further expanded on in the appendix. Furthermore, despite necessitating the usage of a frozen text-image critic model, all three pieces of the objective are relatively cheap to compute compared to the rest of the training procedure, given that they are computed mainly on the forward passes, with the backward pass required for gradient matching being limited to a thin linear probe downstream of the vision encoder.

S2: The results reported by the authors are strong, as they seem to suggest the establishing of a new Pareto frontier on the amount of compute. It is particularly remarkable to note that the framework proposed by the authors is not only limited to dataset compression, indeed it seems that generating synthetic data using GMGM produces higher quality text-image pairs than the raw data available. Since there is no limit on the amount of synthetic data that can be generated (as opposed to the finite quantity of samples inherent to any curated dataset), training can be scaled further without collapse in model quality.

Critical weaknesses

W1: Previous work [1] has established the existence of a modality gap in multi-modal self-supervised representation learning models. This gap consists of the misalignment of identical semantical concepts when embedded through e.g. the text or image modality. In practice, the vector embedding the word "car" and an image of a car might reside in separate regions of the learned latent space. Recent work [2] has shown that the misalignment introduced by this modality gap can be reduced, due to it being largely approximated by an orthogonal transformation (in other words, there seems to be an inherent rotation that differentiates between input modalities, and this rotation can be learned and inverted given a batch of data).

In this work, semantic alignment across modalities is one of the three properties that is sought to be preserved by the proposed GMGM objective. Despite this, the authors fail to address this modality gap in their proposed method (Equation 2), despite correctly identifying its presence in the form of the δreal\delta_{real} component of LalignL_{align}. By performing the GMGM update on a mini-batch of prompt-image pairs, or by generating a synthetic minibatch of pairs for each input, the modality gap could be successfully bridged, potentially resulting in decreases in hallucination rate and bias.

W2: One of the central claims of the paper is the reduction of bias inherent in large-scale T2I datasets. Part of this claim is substantiated by the reduced tendency for Hallucinations (Fig. 2a) and the higher level of semantic grounding (Fig. 2b and 3) induced by GMGM finetuning. In the ethics statement and Appendix, they acknowledge that a biased critic might carry his bias over to the GMGM finetuned model. The impact of the chosen reference data is, however, not discussed. Since distillation is well known to amplify model biases[3], and given the potential impact of this work on the field of Generative Models, care should be taken to reduce this tendency as much as possible.

[1] Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. 2022. Mind the gap: understanding the modality gap in multi-modal contrastive representation learning. In Proceedings of the 36th International Conference on Neural Information Processing Systems (NIPS '22). Curran Associates Inc., Red Hook, NY, USA, Article 1280, 17612–17625.

[2] Antonio D’Orazio, Maria Rosaria Briglia, Donato Crisostomi, Dario Loi, Emanuele Rodolà, and Iacopo Masi. 2026. Implicit Inversion turns CLIP into a Decoder. In The Fourteenth International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=hvukI2ws5O

[3] Jaimeen Ahn, Hwaran Lee, Jinhwa Kim, and Alice Oh. 2022. Why Knowledge Distillation Amplifies Gender Bias and How to Mitigate from the Perspective of DistilBERT. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 266–272, Seattle, Washington. Association for Computational Linguistics.

Questions for authors

Q1: Can the authors provide a small-scale study on the presence of a modality gap in the encoded latents, as well as a short ablation showing the potential impact of modality gap removal on the performance of the method? Given the already impressive results in terms of semantic grounding (Fig. 3), the removal of the modality gap could further tighten the upper bound on training risk by shrinking the δreal\delta_{real} component, which is currently irreducible by sole optimization of LalignL_{align}.

Q2: Can the authors comment on the potential impact of a biased reference dataset? In case of presence of potential bias induced by reference data, can the Ethics Statements be extended to include some comments on the existence of this risk, as well as potential mitigations? Do the authors think that, within the time constraints of the short rebuttal period, it would be reasonable to do a qualitative study on the potential biased introduced by GMGM post-distillation, in a similar fashion to the current manuscript's Figure 3?

5 comments

We appreciate your constructive feedback and the insightful connections to prior work on the CLIP modality gap and bias amplification in distillation. All corresponding updates have been incorporated into the revised manuscript and appendix (highlighted in red in the updated PDF); throughout this response, tags such as [MAIN-P5] and [APP-P24] denote continuous PDF page numbers in the main paper and appendix, respectively.

AuthorsOpenReview ↗

W1 + Q1: Modality-gap diagnosis and calibration

Motivated by Liang et al. (2022) and the orthogonal alignment formulation of D'Orazio et al. (2026, your reference [2]), we measured the image-text modality gap on held-out pairs and tested whether calibrating the global orthogonal rotation between CLIP image and text embeddings improves GMGM synthesis and downstream fine-tuning.

On 4,000 training-domain image-text pairs (deduplicated at the image level and disjoint in prompt templates from the 1,000 evaluation pairs and all downstream benchmarks), we solved the orthogonal Procrustes problem over unit-normalized embeddings h^i\widehat h_i and u^i\widehat u_i:

\sum_{i\in\mathcal C_{\mathrm{fit}}} \\|Q\widehat h_i-\widehat u_i\\|_2^2.$$ Freezing $Q^*$ during synthesis, we replaced Eq. 2 with $$L_{\mathrm{align}}^{Q}(z;t) =1-\cos\\!\left(Q^*h(\mathcal G(z;t)),u(t)\right),$$ while holding the generator, SAE inputs, proxy-gradient objective, prompts, initial latents, and reference batches unchanged. ### Held-out geometry | Alignment representation | Mean real-pair cosine distance | Image/text mean separation | Image-to-text / text-to-image retrieval | |---|---:|---:|---:| | Original CLIP alignment | 0.682 | 0.61 | 0.21 / 0.19 | | Orthogonally calibrated alignment | 0.541 | 0.28 | 0.29 / 0.27 | Evaluated on the same 1,000 disjoint held-out pairs (where mean separation is the cosine distance between modality centroids and retrieval is bidirectional Recall@1), orthogonal calibration cuts centroid separation in more than half (0.61 to 0.28) and lifts Recall@1 by +8 percentage points in both directions. ### Matched synthesis and fine-tuning We then modified only Eq. 2 in a matched 5K synthesis and fine-tuning ablation (SDXL-1.0, $K=50$ latent steps), evaluating image-instruction consistency with the external Qwen2.5-VL-72B judge: | GMGM semantic alignment | Winoground | POPE F1 | MMBench | Independent image-instruction consistency | |---|---:|---:|---:|---:| | Original | 12.8 | 85.1 | 62.8 | 0.82 | | Calibrated | 13.4 (+0.6) | 85.6 (+0.5) | 63.4 (+0.6) | 0.85 (+0.03) | Shrinking the modality gap yields consistent gains across all three downstream benchmarks (+0.6 / +0.5 / +0.6 points) and raises external image-instruction consistency by +3 percentage points. In the theoretical analysis, this calibration acts directly on the real-pair alignment residual $\delta_{\mathrm{real}}=\mathbb E_{(x_r,t)\sim P_{\mathrm{real}}}[1-\cos(h(x_r),u(t))]$ in Lemma 1. Optimizing synthetic latents $z$ lowers $L_{\mathrm{align}}$ while leaving real embeddings untouched, whereas fitting $Q^*$ reduces $\delta_{\mathrm{real}}$ on held-out pairs by 0.141 (from 0.682 to 0.541). In our submitted proof (Theorem 2, Step 1), $\delta_{\mathrm{real}}$ was absorbed into the optimal joint task error $\lambda_{\mathrm{opt}}^*$. `[MAIN-P5]`, `[APP-P14]`, `[APP-P15]`, `[APP-P24]`, and `[APP-P25]` Revised Lemma 1 (Appendix B.2) retains $\delta_{\mathrm{real}}$ and its calibrated counterpart $\delta_{\mathrm{real}}^Q$ explicitly in the paired embedding bound $\mathbb E\\|h(x_s)-h(x_r)\\|_2 \le \sqrt{2\mathbb E[L_{\mathrm{align}}]} + \sqrt{2\delta_{\mathrm{real}}}$, keeping this modality-gap residual separate from $\lambda_{\mathrm{opt}}^*$, while Sec. 3.2 and Appendix J report the calibration ablation. For the complementary RKHS decomposition of our fixed-reference SAE objective, please also see our response to **Reviewer Hwax, W1 + Q1: What the singleton objective controls**.
AuthorsOpenReview ↗

W2 + Q2: Reference-induced scene bias and mitigation

Although reference images in Breal\mathcal B_{\mathrm{real}} are never used directly in MLLM fine-tuning, they shape synthesized training data through both the SAE feature target (Eq. 3) and the mean proxy-gradient target (Eq. 5). Skew in the reference pool can therefore propagate into generated images and persist after fine-tuning, analogous to bias amplification in distillation (Ahn et al., 2022, your reference [3]).

GMGM draws 50 reference images per super-category from the ImageNet-21k training split, whose annotations provide object synsets without scene labels. To run a controlled intervention within the rebuttal window that quantifies bias propagation, we labeled candidate reference images and all generated images using a Places365 classifier (Zhou et al., 2018) with its official indoor/outdoor scene hierarchy and top-10 voting rule. Within a super-category group spanning both contexts, we drew 50 references per condition from a shared pool (an indoor-skewed 45:5 split vs. a balanced 25:25 split) alongside passive generation, holding prompts, initial latents, synthesis settings, and the LoRA recipe fixed. We audited generated scene shares across 200 images (100 probe prompts ×\times 2 latent seeds) and evaluated post-fine-tuning indoor/outdoor scene-classification accuracies on a balanced held-out test set:

Reference condition (indoor:outdoor)Generated indoor shareIndoor / outdoor accuracyGroup gap
Indoor-skewed (45:5)86%73% / 56%17 pp
Rebalanced (25:25)49%70% / 64%6 pp
Passive generation (no references)47%69% / 58%11 pp

Under a 90% indoor reference batch (45:5), prompts without explicit scene constraints default heavily to indoor backgrounds (86% indoor share vs. 47% under passive generation), creating a 17 percentage-point downstream accuracy gap (73% indoor vs. 56% outdoor). Stratifying the reference batch to 25:25 restores balanced generation (49% indoor) and narrows the downstream subgroup gap from 17 to 6 percentage points, lifting outdoor accuracy from 56% to 64% while indoor accuracy shifts from 73% to 70%. [APP-P25] Appendix K details this controlled intervention and its labeling and evaluation protocol.

[MAIN-P11] We have also expanded the Ethics Statement with the following paragraph:

Skew in the real reference batch can also propagate through the SAE feature and mean proxy-gradient anchors (Eqs. 3 and 5), biasing synthesized scene distributions and downstream error rates even though reference images are never used directly in fine-tuning. In a controlled indoor-outdoor case study (Appendix K), rebalancing a skewed 45:5 reference split to 25:25 narrows the post-fine-tuning group-accuracy gap from 17 to 6 percentage points. When reliable attribute labels and sufficient images from each subgroup exist within a category, we recommend stratified reference sampling alongside subgroup accuracy reporting; when subgroup representation in the pool is sparse, practitioners should expand reference coverage before synthesis.

References:

  • Liang et al. (2022). Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. NeurIPS.
  • D'Orazio et al. (2026). Implicit Inversion Turns CLIP into a Decoder. ICLR.
  • Ahn et al. (2022). Why Knowledge Distillation Amplifies Gender Bias and How to Mitigate from the Perspective of DistilBERT. GeBNLP Workshop, ACL, pp. 266-272.
  • Zhou et al. (2018). Places: A 10 Million Image Database for Scene Recognition. IEEE TPAMI, 40(6), 1452-1464.
Reviewer_9Px2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Generative Multi-modal Gradient Matching (GMGM), which optimizes latent codes of a frozen text-to-image generator to create images for downstream MLLM fine-tuning. Its objective combines prompt alignment, sparse-autoencoder feature-distribution matching to a small real reference set, and gradient matching through a proxy learner. The authors connect the three terms to a generalization-gap bound and evaluate synthetic-data fine-tuning across multimodal benchmarks and model backbones, reporting gains over prompt-based generation such as TreeSynth and improved data efficiency.

Critical strengths
  1. The paper identifies a distinction between images that look plausible and images that provide useful training signals. Combining feature coverage with gradient information is a principled attempt to address this gap, rather than relying on prompt diversity alone.
  2. The evaluation includes multiple task benchmarks, data budgets, ablations, and cross-backbone experiments. The comparisons with TreeSynth and with feature-only/gradient-only variants provide useful evidence about the proposed ingredients.
Critical weaknesses
  1. The gradient objective relies on a randomly initialized proxy probe and a small real reference set, while the claimed downstream utility concerns fine-tuning substantially different MLLMs. The theoretical bound does not by itself establish that matching this proxy's gradients predicts the downstream model's learning dynamics. The method's main novelty and generality therefore hinge on the empirical validity of this proxy relationship, which is not yet convincingly established across task distributions or stronger shifts.
  2. The optimized samples are shaped by a frozen generator, vision-language critic, and SAE, so the method inherits their feature blind spots and may optimize critic-specific artifacts. The reported gains over TreeSynth are promising, but do not fully separate the value of gradient matching from reference-driven feature steering, generator capacity, and optimization compute. This makes the broad claim of efficient, generally useful synthetic data less secure. Prior dataset-distillation work already uses gradient matching, while TreeSynth targets diverse synthetic data; GMGM's claimed advance is their combination with latent optimization and SAE control, making clean proxy and compute controls particularly important.
Questions for authors
  1. Can you show whether the proxy-gradient score predicts downstream fine-tuning gains on held-out prompts, categories, and model architectures, including cases where the proxy and target MLLM differ substantially?
  2. Do the reported compute-efficiency comparisons include latent optimization, critic/SAE setup, and downstream training costs for each baseline? A matched end-to-end budget comparison could materially change the efficiency assessment.
6 comments

We appreciate your thoughtful feedback on separating visual plausibility from training utility, as well as your constructive questions on proxy generality and end-to-end compute efficiency. All corresponding updates have been incorporated into the revised manuscript and appendix (highlighted in red in the updated PDF); throughout this response, tags such as [MAIN-P7] and [APP-P22] denote continuous PDF page numbers in the main paper and appendix, respectively.

AuthorsOpenReview ↗

W1 + Q1: Does the proxy predict downstream learning utility?

Because the randomized linear proxy operates on frozen CLIP ViT-L/14 features whereas downstream MLLMs adapt LoRA weights across different architectures, we tested directly whether proxy scores predict realized single-step LoRA validation-loss reduction on both LLaVA-1.6-7B and InternVL2-8B (which pairs a non-CLIP InternViT-300M vision tower with InternLM2). For each held-out candidate with nonzero gradient, we restored an identical warmed-up checkpoint and optimizer state within the target model, applied one LoRA update, and measured validation-loss reduction on a disjoint real split. Incremental ΔR2\Delta R^2 from out-of-sample nested regressions quantifies the predictive variance added by the proxy beyond CLIP alignment, SAE feature scores, and dense-mean cosine similarity combined.

Held-out settingTarget modelProxy SpearmanDense-mean SpearmanIncremental ΔR2\Delta R^2
New prompts/compositions, seen categoriesLLaVA-1.6-7B0.410.26+0.09
New prompts/compositions, seen categoriesInternVL2-8B0.280.19+0.05
New categories, same-category referencesLLaVA-1.6-7B0.330.22+0.06
New categories, same-category referencesInternVL2-8B0.210.17+0.02
New categories, nearest-category referencesLLaVA-1.6-7B0.140.13+0.01

On unseen prompts and compositions within seen categories, the proxy predicts target validation-loss reduction substantially better than dense-mean matching on both LLaVA-1.6-7B (ρ=0.41\rho=0.41 vs. 0.260.26, ΔR2=+0.09\Delta R^2=+0.09) and InternVL2-8B (ρ=0.28\rho=0.28 vs. 0.190.19, ΔR2=+0.05\Delta R^2=+0.05). Under category shift, supplying reference images from the unseen category retains positive incremental signal on LLaVA-1.6-7B (ΔR2=+0.06\Delta R^2=+0.06) and a modest increment on InternVL2-8B (ΔR2=+0.02\Delta R^2=+0.02), whereas substituting nearest-category references lowers ΔR2\Delta R^2 to +0.01+0.01 (ρ=0.14\rho=0.14 vs. 0.130.13), identifying reference-anchor relevance as the boundary condition for category transfer.

This single-step signal carries over to full multi-step fine-tuning. In our 5K ranked-subset study ([APP-P17] Appendix C, Table A3), fine-tuning on top-quartile candidates ranked by proxy score reaches 14.5 / 86.8 / 64.6 on Winoground / POPE F1 / MMBench, outperforming random selection (12.8 / 85.1 / 62.8) and bottom-quartile selection (11.0 / 82.5 / 60.8) under identical data and training compute. [MAIN-P8], [MAIN-P9], and [APP-P17] Main-text Table 4(a) and Appendix C.1 report this two-backbone realized-utility study and full protocol.

AuthorsOpenReview ↗

W2: Separating the proxy from feature steering and generator strength

To isolate randomized proxy-gradient feedback from reference-driven dense feature steering and optimization steps, we synthesized three matched 5K datasets with SDXL-1.0 using identical prompts, initial latents, reference batches, and K=50K=50 AdamW steps: P0 (alignment + SAE regularization), P1 (P0 + cosine matching to the real dense-feature mean), and P2 (P0 + randomized proxy-gradient matching, full GMGM). On an independent development split, P1's weight was calibrated to match the initial latent-gradient norm of P2's proxy term. Each fixed dataset fine-tuned both LLaVA-1.6-7B and InternVL2-8B across three LoRA seeds.

TargetSynthesis objectiveWinogroundPOPE F1MMBench
LLaVA-1.6-7BP0 · Alignment + SAE11.884.661.9
LLaVA-1.6-7BP1 · Dense-mean control12.485.062.4
LLaVA-1.6-7BP2 · Proxy gradient12.8 (+0.4)85.1 (+0.1)62.8 (+0.4)
InternVL2-8BP0 · Alignment + SAE13.985.766.8
InternVL2-8BP1 · Dense-mean control14.686.567.5
InternVL2-8BP2 · Proxy gradient15.1 (+0.5)86.9 (+0.4)68.4 (+0.9)

Parentheses report P2 - P1. Dense-mean matching (P1) captures part of the improvement over P0, confirming that first-order feature steering toward real references is a competitive baseline. Under matched initial gradient norms and K=50K=50 optimization steps, randomized proxy gradients (P2) improve mean scores further across every metric on both backbones, including a statistically separated +0.9 [0.4, 1.4] paired 95% CI gain on InternVL2-8B MMBench.

Base generator capacity is decoupled separately by the crossed 2×22\times 2 matrix in Table 3 (SDXL-1.0 vs. FLUX.2 crossed with passive vs. GMGM). On Qwen3-VL-7B, SDXL + GMGM (21.5 / 89.6 / 71.8) surpasses passive FLUX.2 sampling (18.2 / 87.1 / 69.4) by +3.3 / +2.5 / +2.4 points, and GMGM further boosts FLUX.2 to 26.8 / 91.5 / 74.5. [MAIN-P8], [MAIN-P9], [APP-P17], and [APP-P18] Main-text Table 4(b) and Appendix C.2 present the matched dense-mean control and paired confidence interval.

Reviewer_HwaxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GMGM turns a frozen T2I generator into an active synthetic-data source for MLLM fine-tuning. Per prompt it optimizes the latent with three terms — CLIP text–image cosine alignment (Eq. 2), an SAE-space MMD match to a real reference batch (Eq. 3), and cosine alignment of a randomly initialized linear probe's gradient to the real-batch mean gradient (Eqs. 4–6) — then LoRA-tunes a downstream MLLM on the resulting 50k images. It adds a three-term generalization bound (Thm. 1) and evaluates on six benchmarks over four VLM backbones, crossing SDXL-1.0/FLUX.2 with passive/GMGM synthesis.

Critical strengths
  1. The crossed generator × method design is the paper's most convincing evidence. In Table 3's C0–C4 matrix, SDXL+GMGM (C3: 21.5/89.6/71.8) beats FLUX.2 passive synthesis (C2: 18.2/87.1/69.4), ruling out "a stronger generator explains it". Without this the result would reduce to FLUX.2 > SDXL.
  2. Table 2 answers the cheapest objection. Config A (alignment-only latent optimization) scores 11.5/82.1/61.0, indistinguishable from SDXL-Base (10.9/81.9/60.5), so optimizing z alone does not produce the gains — the added SAE and gradient terms do. That is exactly the control the mechanism needs.
Critical weaknesses
  1. Eq. 3 is not the quantity Theorem 1 needs, and the theorem is never tested. The synthetic side of the MMD is a single sample s(G(z;t)), so only the cross term −2E[k(s,r)] depends on it; the repulsion term E[k(s,s')] is absent, which makes L_feat a kernel-mean attractor toward B_real — the mode-seeking §3.2 explicitly claims to avoid — and nothing bounds ΔS², i.e. Term II. Separately, C1–C3 are never instantiated, L_align/ΔS²/E[L_grad] never measured after optimization, and λ*opt does not depend on Dsyn, so "provably reduces the synthetic-to-real gap" (§1, §3.3) does not follow. Contribution #2 and the stated rationale for the SAE term go with it.
  2. The headline margin is computed against a baseline the paper's own Table 1 shows is near the bottom. TreeSynth (12.4/83.8/62.6) is beaten on NAvg by SRe2L, ShareGPT4V, LGM and even the authors' own Feat-Steering ablation (117.31), yet the abstract's "+5.6% POPE / +58.1% Winoground" uses only TreeSynth. Against Feat-Steering the margin is +1.0 F1 (88.5 vs 87.5) and +21.0% Winoground. Worse, Table 2 Config B (no L_grad) already reaches POPE 88.1 vs Config H's 88.5, so ≈91% of the abstract's POPE gain comes from L_feat, contradicting §4.2's "this isolates the residual gain as gradient informativeness". All LLaVA rows are single-seed (CIs only for Qwen3-VL), and 12.4→19.6 on Winoground's 400 items is ≈29 examples.
Questions for authors
  1. In Eq. 3 the synthetic side has one sample: is E[k(s,s')] included anywhere? If not, how does L_feat deliver concept-level coverage rather than collapse toward the B_real kernel mean, and what bounds Term II?
  2. Report measured L_align, ΔS² and E[L_grad] per Config A–H, numeric C1–C3, and whether the bound ranks the observed gaps. If λ*opt dominates, what does Theorem 1 constrain about the reported deltas?
  3. Seed counts and 95% CIs for Table 1, Table 2 and Fig. 2 on LLaVA-1.6-7B; is Config H's POPE 88.5 distinguishable from Config B's 88.1?
5 comments

We appreciate your close reading of our experimental controls and theoretical formulation, which helped sharpen both our objective-level analysis and multi-seed empirical evaluation. All corresponding updates have been incorporated into the revised manuscript and appendix (highlighted in red in the updated PDF); throughout this response, tags such as [MAIN-P4] and [APP-P15] denote continuous PDF page numbers in the main paper and appendix, respectively.

AuthorsOpenReview ↗

W1 + Q1: What the singleton objective controls

Eq. 3 optimizes each prompt independently with a singleton synthetic feature s(G(z;t))\\{s(\mathcal{G}(z;t))\\}, so the cross-sample kernel term E[k(s,s′)]\mathbb{E}[k(s,s')] does not appear during synthesis. Under a Gaussian kernel with k(s,s)=1k(s,s)=1, only the synthetic-to-reference similarities vary with zz.

At the distribution level, this per-prompt objective upper-bounds set-level discrepancy to the reference batch. Let P^R\widehat P_R denote the empirical reference distribution, QQ the generated SAE-feature distribution, ψ\psi the canonical feature map into RKHS H\mathcal H, and μQ=EQ[ψ(s)]\mu_Q=\mathbb E_Q[\psi(s)]. Expanding ∣ψ(s)−μR∣H2\\|\psi(s)-\mu_R\\|_{\mathcal H}^2 around μQ\mu_Q yields the exact identity

=\mathrm{MMD}_k^2(Q,\widehat P_R) +\mathbb E_Q\\|\psi(s)-\mu_Q\\|_{\mathcal H}^{2}.$$ Because the within-$Q$ RKHS variance term is nonnegative, minimizing expected singleton loss upper-bounds set-level $\mathrm{MMD}_k^2(Q,\widehat P_R)$. Against the population real feature distribution $P$, the triangle inequality in $\mathcal H$ gives $$\mathrm{MMD}_k(Q,P) \le \sqrt{\mathbb E_Q\bigl[L_{\mathrm{single}}(s;R)\bigr]} +\mathrm{MMD}_k(\widehat P_R,P),$$ where $\mathrm{MMD}_k(\widehat P_R,P)$ captures finite-reference sampling error. Our submitted Lemma 2(i) and Theorem 2 (Step 2) conflated the per-image loss directly with population $\Delta_S^2$. `[MAIN-P4]`, `[MAIN-P5]`, and `[APP-P15]` We have corrected this step in Theorem 1 (Sec. 3.3) and Appendix B.3, framing Eq. 3 as reference-based SAE feature regularization. Because each trajectory remains anchored to a distinct prompt $t$ via $\mathcal{L}_{\mathrm{align}}(z;t)$ ($\lambda_{\mathrm{align}}=10$ vs. $\lambda_{\mathrm{feat}}=1$) from independent initial latents $z_0\sim\mathcal{N}(0,I)$, whether Eq. 3 induces mode collapse or expands feature support is an empirical question that we tested directly. We regenerated matched 5K datasets while varying only the feature objective, measuring set-level SAE $\mathrm{MMD}^2$ on a real holdout alongside cluster coverage (512 real-data $k$-means clusters) and near-duplicate rate (pairwise cosine $>0.95$) in an independent DINOv2-L embedding space unseen during synthesis. | Feature objective | Set-level MMD² on real holdout | Independent feature coverage | Duplicate rate | Winoground | POPE F1 | MMBench | |---|---:|---:|---:|---:|---:|---:| | No feature term | 0.0186 | 58% | 24% | 11.9 | 83.9 | 62.1 | | Original singleton objective | 0.0069 | 66% | 13% | 12.8 | 85.1 | 62.8 | | Joint-batch variant | 0.0048 | 73% | 8% | 13.1 | 85.3 | 63.2 | Relative to omitting $L_{\mathrm{feat}}$, the singleton objective lowers holdout SAE $\mathrm{MMD}^2$ from 0.0186 to 0.0069, raises independent DINOv2 coverage by 8 percentage points, cuts near-duplicates by 11 points, and improves all three downstream benchmarks. Adding explicit cross-image repulsion in a joint-batch ablation lifts coverage further to 73% (8% duplicates) with modest additional downstream gains, at the cost of coupling prompts across batch updates. `[APP-P25]` and `[APP-P26]` Appendix L documents this regenerated ablation and its independent coverage metrics.
AuthorsOpenReview ↗

Q2: Measured terms, constants, and the role of the joint error

To examine how the three objective quantities track real-to-synthetic risk gaps, we sampled matched 5K subsets from the original 50K pools of Configs A to H (distinct from the freshly regenerated 5K ablation above, hence ΔS2=0.0071\Delta_S^2=0.0071 vs. 0.00690.0069 for Config H) and evaluated all three terms on a shared real holdout and a common fixed linear probe, including terms whose weights were zero during synthesis.

We instantiate the coefficient expressions from the submitted Theorem 1 using empirical measurements on the diagnostic split:

CoefficientSubmitted Theorem 1 formulaMeasurement / normalizationValue used
C1C_12MBhBu2MB_hB_uM=1M=1, unit-normalized embeddings Bh=Bu=1B_h=B_u=12.00
C2C_2MLS2ML_S^2Empirical SAE encoder spectral norm 1.401.96
C3C_32MBg2MB_gMaximum proxy-gradient norm on diagnostic split, 0.420.84

Here LS=1.40L_S=1.40 and Bg=0.42B_g=0.42 serve as empirical plug-in quantities scoped to the diagnostic split and sampled probes (as opposed to global analytical bounds over all Top-KK support switches). Using these weights without the synthesis hyperparameters λ\lambda, we compare S=C1Lalign+C2ΔS2+C3E[Lgrad]S=C_1L_{\mathrm{align}}+C_2\Delta_S^2+C_3\mathbb E[L_{\mathrm{grad}}] against the validation cross-entropy gap (real minus synthetic) on the fixed probe:

ConfigMean LalignL_{\mathrm{align}}Set-level ΔS2\Delta_S^2E[Lgrad]\mathbb E[L_{\mathrm{grad}}]Weighted sum SSReal - synthetic probe loss
A0.610.02120.441.6310.34
B0.630.00760.411.6190.25
C0.620.01410.421.6200.31
D0.660.00550.451.7090.33
E0.620.01900.191.4370.28
F0.610.02010.261.4780.30
G0.650.01940.151.4640.32
H0.640.00710.171.4370.20

Across Configs A to H, SS correlates positively with the observed probe-loss gap at Spearman ρ=0.738\rho=0.738 (computed from unrounded SS, where E and H evaluate to 1.4368401.436840 and 1.4367161.436716 before three-decimal rounding). Moving from Config A to Config H lowers SS from 1.631 to 1.437 and narrows the probe-loss gap from 0.34 to 0.20, driven primarily by C3E[Lgrad]C_3\mathbb E[L_{\mathrm{grad}}] (−0.227-0.227), alongside −0.028-0.028 from C2ΔS2C_2\Delta_S^2 and +0.060+0.060 from C1LalignC_1L_{\mathrm{align}}. Because C2ΔS2C_2\Delta_S^2 carries small numerical weight in SS, pairwise inversions remain: Config B has a higher SS than Config G (1.619 vs. 1.464) yet a smaller probe-loss gap (0.25 vs. 0.32), and Config H achieves a substantially smaller gap than Config E (0.20 vs. 0.28) at nearly equal SS.

Regarding λopt∗=inf⁡f∈F[R(f;Preal)+R(f;Psyn)]\lambda_{\mathrm{opt}}^* = \inf_{f\in\mathcal{F}}[R(f;P_{\mathrm{real}})+R(f;P_{\mathrm{syn}})], its value depends on PsynP_{\mathrm{syn}} and can vary across Configs A to H, while downstream MLLMs adapt LoRA weights beyond a frozen linear probe. Because an uncontrolled λopt∗\lambda_{\mathrm{opt}}^* prevents a global domain-adaptation bound from certifying downstream benchmark deltas directly, [MAIN-P5], [APP-P26], and [APP-P27] revised Sec. 3.3 replaces the submitted global risk bound with exact objective-level decompositions (Theorem 1 and Lemmas 1 to 3) paired with empirical downstream evaluation, and Appendix M reports the plug-in weights and A to H diagnostic table.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.