acceptodds
Under review as a conference paper at ICLR 2027

Trust What You Match: Reliable Cross-Modal Coupling Matching for Multimodal Dataset Distillation

11 views

Abstract

Multimodal dataset distillation (MDD) aims to compress large-scale corpora into compact synthetic data while preserving their downstream utility. However, existing methods typically match cross-modal couplings estimated from stochastic mini-batches, making supervision highly sensitive to batch composition and consequently impairing preservation of dataset-level multimodal structure. To address this limitation, we propose Reliable Cross-Modal Coupling Matching (ReCoup), which calibrates encoder-space couplings using fluctuations induced by random image–text pairings within each mini-batch. Accordingly, ReCoup assigns higher weights to couplings that are more reproducible across batches and thus better reflect full-dataset statistics. The resulting reliability-aware objective yields more coherent synthetic-data gradients and reduces conflicting optimization updates, while an auxiliary constraint further preserves the diagonal coupling profile in the projected shared space. Experiments on Flickr30K and MS-COCO show consistent improvements over state-of-the-art MDD methods across synthetic-set budgets. With 500 synthetic pairs, ReCoup improves mean Recall by 7.3 points on Flickr30K and 5.1 points on MS-COCO.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multimodal Dataset Distillation, Cross-Modal Coupling Matching, Reliability-Aware Matching, Vision-Language Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ubCtOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper is a work on image-text dataset distillation. It first identify the bottleneck of previous distillation methods, which is the sample-wise similarities in a mini-batch is not reliable during distillation. The paper provide a dimension-wise matching criteria to address this issue, and propose DDCM to further improve the performance.

Critical strengths

The paper is well motivated. The problem is clearly stated and FACM is designed targeting to the problem.

Critical weaknesses
  1. The discussion of computation cost is insufficient. The GPU hours and GPU memory usage should be reported.
  2. d_p=2304 seems to require a huge computation cost. With the same computational budget, sample-wise similarities could be computed on more samples, which may mitigate the sensitive issue introduced by "mini-batch". Can author provide a comparison or analysis on this?
  3. The investigation of hyperparameters is not established. What's the effectiveness of \rho, \gamma, \lambda_{DDCM}
Questions for authors

In the first visualization of the distilled data, it shows a man in white shirt, but the caption is "a man in a blue shirt works on a building". What is the insight of this inconsistency on the distilled data?

5 comments

We appreciate the reviewer's thoughtful comments and address the points below.


Q1 (Weakness 1): The discussion of computation cost is insufficient. The GPU hours and GPU memory usage should be reported.

A1: We thank the reviewer for this comment. We report a computational cost comparison on MS-COCO with 500 synthetic pairs using a single NVIDIA H100 GPU 96GB96 GB.

Table R1. Computational cost comparison on MS-COCO with 500 synthetic pairs.

MethodExpert storageExpert preparationPeak GPU memoryDistillation time
LoRS37.3 GB136.1 h58.4 GB10.1 h
RepBlend16.6 GB77.8 h27.3 GB4.0 h
CovMatch——36.6 GB4.8 h
ReCoup——35.9 GB3.5 h

ReCoup completes distillation in 3.5 GPU-hours with 35.9 GB peak GPU memory, compared with 4.8 GPU-hours and 36.6 GB for CovMatch, reducing distillation time by 27.1%. ReCoup also requires no expert preparation or checkpoint storage, unlike LoRS and RepBlend, which incur 136.1 and 77.8 hours of additional expert preparation, respectively.

These results show that ReCoup achieves its retrieval improvements with lower distillation time and comparable peak GPU memory relative to CovMatch.

We have revised Section 4.4 (Table 2), with changes highlighted in blue.

Q2 (Weakness 2): d_p=2304 seems to require a huge computation cost. With the same computational budget, sample-wise similarities could be computed on more samples, which may mitigate the sensitive issue introduced by "mini-batch". Can author provide a comparison or analysis on this?

A2: We thank the reviewer for this suggestion. Although the projection dimension is dp=2304d_p=2304, DDCM matches only the diagonal of the projected coupling matrix, requiring O(ndp)O(nd_p) computation and O(dp)O(d_p) storage for the matching statistics. FACM computes dense encoder-space couplings with O(ndvdt)O(nd_vd_t) complexity, independent of dpd_p, whereas computing dense sample-wise similarities requires O(n2dp)O(n^2d_p). Larger real-data batches also increase feature extraction costs through the encoders and projection heads. We therefore compare retrieval performance and total distillation time to examine whether larger sample-wise batches can close the performance gap.

We conduct experiments on Flickr30K with 100 synthetic pairs. For sample-wise matching, we use quantile-based MSE to match the positive- and negative-pair distributions of projected image–text cosine similarities. FACM + Sample-wise combines FACM with this objective instead of DDCM, while Sample-wise uses only this objective. We first compare methods at a fixed real-data batch size of 128, then increase the batch size for both sample-wise variants.

Table R2. Performance and runtime comparison at a fixed real-data batch size of 128 on Flickr30K with 100 synthetic pairs.

Methoddpd_pmRTime
Sample-wise76843.01.2 h
FACM + Sample-wise76852.11.4 h
FACM76854.51.0 h
FACM + DDCM76855.41.0 h
FACM + DDCM230456.31.0 h

Table R3. Effect of real-data batch size on sample-wise matching dpd_p=768.

MethodMetric128192256320
Sample-wisemR43.043.744.344.6
Sample-wiseTime1.2 h1.6 h2.0 h2.5 h
FACM + Sample-wisemR52.152.853.053.3
FACM + Sample-wiseTime1.4 h2.0 h2.7 h3.3 h

At the same projection dimension dp=768d_p=768 and real-data batch size 128128, FACM + DDCM achieves 55.4 mR in 1.0 hour, outperforming FACM + Sample-wise 521˙mR,14˙hours52\.1 mR, 1\.4 hours by 3.3 points. Increasing the batch size to 320 improves FACM + Sample-wise to 53.3 mR, but also increases its runtime to 3.3 hours. Even with this larger batch, it remains 2.1 points below FACM + DDCM, despite taking 3.3 times as long. Sample-wise alone also improves from 43.0 to 44.6 mR but remains substantially weaker. Increasing dpd_p from 768 to 2304 further improves FACM + DDCM from 55.4 to 56.3 mR, with both configurations taking approximately 1.0 hour.

These results show that larger sample-wise batches do not close the performance gap with FACM + DDCM within the evaluated range, despite requiring more computation.

We have added this analysis in Appendix D.10 (Table 16), with changes highlighted in blue.

Q3 (Weakness 3): The investigation of hyperparameters is not established. What's the effectiveness of ρ\rho, γ\gamma, λ{DDCM}\lambda_\{DDCM\}?

A3: We thank the reviewer for this suggestion. We conducted a sensitivity analysis on Flickr30K with 100 synthetic pairs, varying one parameter at a time while keeping the others at their default values $$ρ=1, γ=2, λ_{DDCM}=0.1$). Results are reported in mean Recall (mR$.

DDCM weight λDDCMλ_{DDCM}. Removing DDCM reduces mR from 56.3 to 54.6 −17˙points−1\.7 points, supporting its complementary contribution to FACM. Performance remains between 55.7 and 56.5 for $\lambda_{DDCM} \in

00˙5,10˙0\.05,1\.0

$, indicating limited sensitivity within this range.

Table R4. Sensitivity to the DDCM weight λ_DDCM.

λDDCMλ_{DDCM}00.010.050.10.20.51
mR54.655.555.856.356.555.855.7

FACM exponent γγ. Increasing γ\gamma from 0 (uniform weighting) to 2 improves mR by 6.5 points, demonstrating the benefit of emphasizing more reliable couplings. Performance slightly declines at γ=4\gamma=4, suggesting diminishing benefits from stronger weight concentration.

Table R5. Sensitivity to the FACM exponent γγ.

γγ00.5124
mR49.851.755.156.355.7

Target scaling ρρ. The default $\rho=1$ achieves the highest mR 563˙56\.3, with nearby values of 0.75 and 1.25 yielding 56.0 and 55.8. Reducing ρ\rho to 0.5 lowers mR to 54.1, indicating that substantial downscaling of real-data coupling targets can impair performance.

Table R6. Sensitivity to the target scaling coefficient ρρ.

ρρ0.50.7511.251.5
mR54.156.056.355.855.4

We have added the sensitivity analysis in Appendix D.6 Fig\. 8, highlighted in blue.


Q4 (Question 1): In the first visualization of the distilled data, it shows a man in white shirt, but the caption is "a man in a blue shirt works on a building". What is the insight of this inconsistency on the distilled data?

A4: We thank the reviewer for this observation. Following the visualization procedure of CovMatch, we retrieve the nearest caption from the corresponding dataset MS−COCOorFlickr30KMS-COCO or Flickr30K for each optimized synthetic text embedding. These captions are used only for visualization, while downstream training directly uses the optimized continuous text embeddings.

In the highlighted example, the retrieved caption captures the main activity but differs in shirt color. This illustrates a limitation of using retrieved captions to interpret distilled image–text pairs: they can convey the overall semantics without accurately capturing every visual detail.

We have clarified this point in Appendix E, with changes highlighted in blue.

Reviewer_KhPiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ReCoup for multimodal dataset distillation. It matches cross-modal coupling matrices between real and synthetic data, using correlation-based weights in the encoder space and an additional diagonal matching objective in the projected space.

Critical strengths
  • The analyses show that couplings with higher weights are consistent across batches and that FACM reduces conflicting updates to synthetic data.

  • ReCoup consistently improves retrieval performance across the evaluated datasets and synthetic pair budgets and shows strong cross-architecture generalization.

Critical weaknesses
  • FACM is not evaluated with smaller batches or compared with unweighted matching to coupling matrices averaged across batches. Its robustness to batch size and advantage over this simpler alternative remain unclear.

  • The paper does not compare the proposed objective with FACM applied only to projected representations and evaluates only BERT as the text encoder. The need for matching at both stages and generalization to other text encoders remain unclear.

Questions for authors
  • The paper does not evaluate sensitivity to real-data batch size. Since FACM relies on correlations estimated within each batch, it remains unclear whether it remains effective with smaller batches that may provide less reliable correlation estimates.

  • Averaging cross-modal coupling matrices across multiple real-data batches could reduce batch fluctuations without reliability weighting. It remains unclear whether FACM outperforms this simpler alternative.

  • The paper applies FACM to encoder outputs and DCM to projected representations. It does not compare this combination with applying FACM only to projected representations.

  • The method is evaluated only with BERT as the text encoder. It remains unclear whether similar performance gains hold with other text encoders.

8 comments

We thank the reviewer for the constructive feedback. Below, we address each comment in turn.


Q1 Weakness1&Question1Weakness 1 \& Question 1: The paper does not evaluate sensitivity to real-data batch size. Since FACM relies on correlations estimated within each batch, it remains unclear whether it remains effective with smaller batches that may provide less reliable correlation estimates.

A1: We thank the reviewer for raising this concern. We conducted an additional sensitivity analysis on Flickr30K with 500 synthetic pairs, varying the real-data batch size from 50 to 300.

Table R1. Sensitivity to real-data batch size on Flickr30K with 500 synthetic pairs.

Real batch sizeUniform CCMUniform CCM+DDCMFACMReCoup (FACM + DDCM)
5037.838.744.044.9
10049.750.655.856.7
15051.052.056.557.6
20052.453.357.658.6
25052.753.658.159.2
30053.254.358.559.7

Across all six batch sizes, FACM consistently outperforms Uniform CCM by 5.2–6.2 mR points. Even at the smallest batch size of 50, FACM achieves 44.0 mR compared with 37.8 for Uniform CCM, while ReCoup further improves performance to 44.9 mR.

Although absolute performance decreases with smaller batches, FACM maintains its advantage over uniform matching, demonstrating that reliability-aware weighting remains effective under smaller real-data batch sizes.

We have added this analysis in Section 4.3 (Fig. 6), highlighted in blue.


Q2 Weakness1&Question2Weakness 1 \& Question 2: Averaging cross-modal coupling matrices across multiple real-data batches could reduce batch fluctuations without reliability weighting. It remains unclear whether FACM outperforms this simpler alternative.

A2: We thank the reviewer for this suggestion. According to Eq. 44, averaging coupling matrices across mm real-data batches of nn samples each is mathematically equivalent to computing a single coupling matrix from a concatenated batch of mnmn samples, provided the model representations remain fixed:

$C_{\mathrm{avg}}=\frac{1}{m}\sum_{k=1}^{m}C^{(k)} =\frac{1}{mn}X^\top Y=C_{\mathrm{large}}.$

Thus, cross-batch averaging reduces sampling fluctuations by using more real samples, whereas FACM weights individual couplings based on their reliability.

We evaluate this alternative on Flickr30K with 100 synthetic pairs. We compare ReCoup FACM+DDCMFACM + DDCM using one real batch of 128 pairs against Uniform CCM + DDCM with coupling matrices averaged over 1–16 real batches, each containing 128 pairs.

Table R2. Comparison with unweighted cross-batch averaging on Flickr30K.

MethodReal batchesEffective batch sizemRTime
Uniform CCM + DDCM112849.81.0h
Uniform CCM + DDCM225650.21.6h
Uniform CCM + DDCM451250.72.9h
Uniform CCM + DDCM8102450.95.5h
Uniform CCM + DDCM16204851.310.7h
ReCoup (FACM + DDCM)112856.31.0h

Averaging 16 batches improves the unweighted baseline from 49.8 to 51.3 mR. In contrast, ReCoup achieves 56.3 mR using a single batch, outperforming the 16-batch alternative by 5.0 points, with substantially lower runtime 1\.0 vs\. 10\.7 hours.

These results show that increasing the number of averaged batches does not match the performance of FACM within the evaluated range. We have added this comparison in Appendix D.7 (Table 12), with the subsection heading highlighted in blue.

Q3 Weakness2&Question3Weakness 2 \& Question 3: The paper applies FACM to encoder outputs and DCM to projected representations. It does not compare this combination with applying FACM only to projected representations.

A3: We thank the reviewer for this suggestion. We conducted an additional experiment on Flickr30K with 100 synthetic pairs, comparing projected-space FACM, encoder-space FACM, and ReCoup encoder−spaceFACM+projected−spaceDDCMencoder-space FACM + projected-space DDCM.

Table R3. Comparison of FACM across representation stages on Flickr30K.

MethodmRPeak memory
Projected FACM only51.544.5 GB
Encoder FACM only54.616.7 GB
ReCoup EncoderFACM+DDCMEncoder FACM + DDCM56.317.0 GB

Encoder-space FACM outperforms projected-space FACM by 3.1 mR points, while reducing peak memory from 44.5 to 16.7 GB. Adding projected-space DDCM further improves mR by 1.7 points, with only 0.3 GB additional memory.

These results support using FACM to match dense cross-modal couplings in the encoder space and DDCM to preserve the diagonal coupling structure in the projected space.

We have added this comparison in Appendix D.8 (Table 13), highlighted in blue.


Q4 Weakness2&Question4Weakness 2 \& Question 4: The method is evaluated only with BERT as the text encoder. It remains unclear whether similar performance gains hold with other text encoders.

A4: We thank the reviewer for this suggestion. We conducted additional experiments on Flickr30K with 100 synthetic pairs, using NFNet+BERT for distillation and NFNet+DistilBERT for evaluation across all methods.

Table R4. Cross-text-encoder generalization from BERT to DistilBERT on Flickr30K (100 synthetic pairs).

MethodmR
MTT-VL20.2
LoRS23.5
CovMatch27.1
ReCoup42.1

ReCoup achieves 42.1 mR, outperforming the strongest baseline (CovMatch) by 15.0 points, with improvements across all six retrieval metrics. These results demonstrate that ReCoup's distilled data generalize to an unseen text encoder.

We have added these results in Appendix D.9 (Table 15), highlighted in blue.

Thank you for the detailed response and the additional experiments. However, several concerns remain. In the cross-batch averaging comparison, enlarging the effective batch to 2048 improves Uniform CCM + DDCM by only 1.5 mR, whereas FACM gains 6.5 mR from a single batch of 128. This suggests that the gain of FACM does not come from reducing batch fluctuations, which conflicts with the motivation of the paper that mini-batch sensitivity is the main limitation of uniform matching. Please clarify what mechanism accounts for the improvement. In addition, the averaging baseline recomputes 16 batches at every iteration, while a running average across iterations would add little cost, so the runtime comparison overstates the overhead of the alternative. For the text-encoder question, the response evaluates transfer from BERT to DistilBERT rather than distillation with a different text encoder, so the original question remains unanswered.

Reviewer_RkqDOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors proposed Reliable Cross-Modal Coupling Matching (ReCoup), aligning encoder-space couplings of the real dataset and synthetic dataset. Specifically two modules are introduced, Fluctuation-Aware Cross-Modal Coupling Matching (FACM) calibrates modality encoder-level coupling supervision against a random-pairing null and DimensionWise Diagonal Coupling Matching (DDCM) put projection space constraint on corresponding cross-modal dimensions. Experiments on Flickr30K and MS-COCO show consistent improvements over state-of-the-art MDD methods across synthetic-set budgets.

Critical strengths

S1. The paper is well written, easy to follow.

S2. Strong empirical results, outperforms baseline by a good margin given constraint budget.

Critical weaknesses

W1. The paper's conclusion is more empircal rather than providing deep understanding to it. Deeper insight of why diagonal coupling and dimension-wise similarity works better is missing. DDCM is not theoretically justified.

W2. The model options tested in the experiments are quite limited. No SoTA/modern model architecture is tested, it's not convinced that the conclusion can hold and generalize.

Questions for authors

Q1. The weights wijw_{ij} and coupling matrices CijC_{ij} are derived using the same batch, one centered and one uncentered. Will this amplify the sampling noise?

Q2. Could the authors provide the computational cost comparison with the baselines (end-to-end approaches) to understand the tradeoffs? Paper includes the DDCM cost but a heavier computation is required for FACM.

Q3. How does the batch size affect the performance gains from each module?

Q4. How robust are the learned representations? Can it generalize to other tasks?

I'm willing to increase my score if my concerns are addressed.

9 comments

We appreciate the reviewer’s thoughtful comments and provide point-by-point responses below.


Q1 (Weakness 1): The paper's conclusion is more empircal rather than providing deep understanding to it. Deeper insight of why diagonal coupling and dimension-wise similarity works better is missing. DDCM is not theoretically justified.

A1: We thank the reviewer for this comment. We provide further mathematical explanations and empirical evidence for why dimension-wise diagonal coupling matching is effective.

For the column-normalized projected representations used by DDCM,

=\operatorname{tr}(C^p) =\sum_{j=1}^{d_p}C^p_{jj}.$$ Thus, each diagonal coupling represents the contribution of a corresponding feature dimension to the sum of paired image–text inner products\. Fig\. 5$a$ shows that the diagonal\-to\-off\-diagonal ratio $DOR$ increases during optimization for paired and jointly rotated representations, but remains near 1 under random pairing\. To further examine the reliability of these associations, we compute the ratio of mean diagonal to mean off\-diagonal FACM\-style reliability weights in the projected space\. This ratio increases from approximately 1 at initialization to over 2 by step 10 and 11\.8 by step 200 $Fig\. 7 in Appendix D\.4$\. Together, these results show that diagonal associations become increasingly prominent in both coupling magnitude and FACM\-style reliability weights\. These observations motivate the use of diagonal matching in distillation\. Let $\Delta=C^{p,s}-\rho C^{p,r}$ denote the projected coupling mismatch\. The uniformly weighted full\-matrix objective can be written as \$\mathcal L\_{\mathrm{full}} =\frac{1}{d\_p^2}\|\Delta\|\_F^2 =\frac{1}{d\_p}\mathcal L\_{\mathrm{DDCM}} +\left(1-\frac{1}{d\_p}\right)\mathcal L\_{\mathrm{off}},\$ where $\mathcal L_{\mathrm{off}}$ is the mean squared mismatch over off\-diagonal entries\. Thus, full\-matrix matching scales the mean diagonal matching error by only $1/d_p$ and includes $d_p(d_p-1)$ off\-diagonal terms\. Our analysis shows that off\-diagonal associations have smaller average magnitudes and lower FACM\-style reliability weights\. Their matching targets are also estimated from finite mini\-batches and can vary with batch composition\. As a result, full\-matrix matching requires the small synthetic set to match not only the diagonal couplings that directly contribute to paired similarity, but also many weaker, batch\-dependent off\-diagonal associations\. DDCM instead focuses on diagonal couplings, which show greater average magnitudes and higher FACM\-style reliability weights $Figs\. 5(a$ and 7\)\. This avoids the additional off\-diagonal matching terms and reduces computation from $O(nd_p^2)$ to $O(nd_p)$\. Matching only the trace, however, loses information about individual dimensions\. Let $e_j=d_j^{p,s}-\rho d_j^{p,r}$ denote the error of the $j$\-th diagonal coupling\. Trace matching constrains only $(\sum_j e_j)^2$, which can vanish even when individual errors are large because positive and negative errors cancel out\. In contrast, DDCM minimizes $\frac{1}{d_p}\sum_j e_j^2$, penalizing each dimension separately and preventing such cancellation\. Consistent with this analysis, DDCM achieves 56\.3 mR, compared with 54\.8 for trace matching and 55\.7 for full\-matrix matching $Fig\. 5(b$\)\. We have added the corresponding analysis in Appendix D\.4, with changes highlighted in blue\.

Q2 (Weakness 2): The model options tested in the experiments are quite limited. No SoTA/modern model architecture is tested, it's not convinced that the conclusion can hold and generalize.

A2: We thank the reviewer for this suggestion. We further evaluate ReCoup against CovMatch using DINOv3 ViT-B/16 20252025 [1] as the visual encoder and UltraBERT-150M 20262026 [2] as the text encoder. Both encoders are independently pretrained without joint image–text alignment and have approximately 0.24B parameters in total.

For each method and encoder configuration, we distill 100 synthetic image–text pairs from MS-COCO and use them to train the corresponding vision–language model. We evaluate the resulting models on MS-COCO and directly on Flickr30K without additional training or adaptation.

Table R1. Cross-architecture comparison on MS-COCO and Flickr30K with 100 synthetic pairs.

Encoder configurationMethodMS-COCOFlickr30K
NFNet + BERTCovMatch21.029.9
NFNet + BERTReCoup25.237.1
DINOv3 + UltraBERTCovMatch26.737.2
DINOv3 + UltraBERTReCoup31.744.8

ReCoup outperforms CovMatch under both encoder configurations. With DINOv3+UltraBERT, the gains reach 5.0 mR points on MS-COCO and 7.6 points on Flickr30K, compared with 4.2 and 7.2 points under NFNet+BERT. These improvements hold across all six retrieval metrics on both datasets, showing that ReCoup maintains its advantage with modern encoders and supports cross-dataset retrieval transfer.

We have added the full retrieval results in Appendix D.9 (Table 14), with changes highlighted in blue.

References

[1] Siméoni O, Vo H V, Seitzer M, et al. Dinov3[J]. arXiv preprint arXiv:2508.10104, 2025.

[2] Park J, Lee T, Kang J. UltraBERT: Beyond the Limit of BERT Pre-Training[J]. arXiv preprint arXiv:2610.15028, 2026.


Q3 (Question 1): The weights WijW_{ij} and coupling CijC_{ij} matrices are derived using the same batch, one centered and one uncentered. Will this amplify the sampling noise?

A3: We thank the reviewer for this question. To examine the effect of same-batch weight estimation, we conduct an additional experiment on Flickr30K with 100 synthetic pairs.

At each distillation step, we independently sample two real-data batches and compute their coupling matrices C1,C2C_1,C_2 and reliability weights W1,W2W_1,W_2. Let Δb,ij=Cijs−ρCb,ij\Delta_{b,ij}=C^s_{ij}-\rho C_{b,ij}. We compare:

\mathcal{L}_{\mathrm{same}} &= \frac{1}{2d_vd_t}\sum_{ij} \left(W_{1,ij}\Delta_{1,ij}^2 + W_{2,ij}\Delta_{2,ij}^2\right), & \mathcal{L}_{\mathrm{cross}} &= \frac{1}{2d_vd_t}\sum_{ij} \left(W_{2,ij}\Delta_{1,ij}^2 + W_{1,ij}\Delta_{2,ij}^2\right). \end{aligned}$$ Cross\-Batch FACM swaps the reliability weights while keeping the coupling targets unchanged\. We also evaluate two\-batch Uniform CCM without reliability weighting\. **Table R2\.** Same\-batch vs\. cross\-batch weighting on Flickr30K with 100 synthetic pairs\. | **Method** | **IR@1** | **IR@5** | **IR@10** | **TR@1** | **TR@5** | **TR@10** | **mR** | | ----------------------- | -------------: | -------------: | --------------: | -------------: | -------------: | --------------: | -------------: | | Uniform CCM (2 batches) | 22.3 | 51.1 | 64.4 | 30.3 | 60.2 | 72.0 | 50.1 | | Cross-Batch FACM | 26.8 | 57.6 | 70.1 | **37.4** | 67.5 | 77.8 | 56.2 | | Same-Batch FACM | **27.2** | **58.1** | **70.3** | 36.6 | **68.2** | **78.6** | **56.5** | Same\-Batch and Cross\-Batch FACM achieve comparable performance $56\.5 vs\. 56\.2 mR$, both substantially outperforming Uniform CCM $50\.1 mR$\. This suggests that estimating weights and couplings from the same batch does not introduce a noticeable performance penalty\. Furthermore, our existing gradient analysis \(Fig. 4(c), with further details in Appendix D.3) shows that FACM improves gradient agreement from 0\.138 to 0\.188 and reduces conflicting gradient pairs from 2\.53% to 0\.23% relative to uniform matching, supporting more consistent optimization signals\. We have added this analysis in Appendix D\.3 \(Table 9\), with changes highlighted in blue\.

Q4 (Question 2): Could the authors provide the computational cost comparison with the baselines to understand the tradeoffs? Paper includes the DDCM cost but a heavier computation is required for FACM.

A4: We thank the reviewer for this suggestion. We compare the computational costs of ReCoup and representative baselines on MS-COCO with 500 synthetic pairs, including expert preparation and distillation.

Table R3. Computational cost comparison on MS-COCO with 500 synthetic pairs.

MethodExpert storageExpert timePeak memoryDistillation time
LoRS37.3 GB136.1 h58.4 GB10.1 h
RepBlend16.6 GB77.8 h27.3 GB4.0 h
CovMatch——36.6 GB4.8 h
ReCoup——35.9 GB3.5 h

FACM requires O(ndvdt)O(nd_vd_t) computation and O(dvdt)O(d_vd_t) stored statistics. Computing its reliability weights adds constant-factor overhead without increasing the asymptotic complexity of dense coupling matching (Appendix B.4, Table 4).

Despite this additional computation, ReCoup achieves the shortest distillation time (3.5 h), with peak memory comparable to CovMatch (35.9 vs. 36.6 GB). Unlike LoRS and RepBlend, ReCoup also requires no expert preparation, avoiding substantial additional training time and storage.

We have revised Section 4.4 (Table 2), with changes highlighted in blue.


Q5 (Question 3): How does the batch size affect the performance gains from each module?

A5: We thank the reviewer for this question. We evaluate FACM and DDCM across real-data batch sizes from 50 to 300 on Flickr30K with 500 synthetic pairs.

Table R4. Sensitivity to real-data batch size on Flickr30K with 500 synthetic pairs.

Real batch sizeUniform CCMUniform CCM+DDCMFACMReCoup (FACM + DDCM)
5037.838.744.044.9
10049.750.655.856.7
15051.052.056.557.6
20052.453.357.658.6
25052.753.658.159.2
30053.254.358.559.7

As the batch size decreases from 300 to 50, absolute mR decreases across all methods, but the improvements from both modules remain substantial. Relative to Uniform CCM, FACM's gain changes from +5.3 to +6.2 mR points, while DDCM's gain changes from +1.1 to +0.9 points. Adding DDCM to FACM also consistently improves performance, with gains of 0.9–1.2 points across the evaluated batch sizes. These results show that FACM maintains its main contribution even with smaller batches, and DDCM provides a consistent complementary improvement.

We have added this analysis in Section 4.3 (Fig. 6), highlighted in blue.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion deadline (November 18, AoE) approaches, please review the authors’ rebuttal if you have not already done so and post a follow-up confirming that you have read it and indicating whether your main concerns have been addressed.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.