- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper is a work on image-text dataset distillation. It first identify the bottleneck of previous distillation methods, which is the sample-wise similarities in a mini-batch is not reliable during distillation. The paper provide a dimension-wise matching criteria to address this issue, and propose DDCM to further improve the performance.
Critical strengths
The paper is well motivated. The problem is clearly stated and FACM is designed targeting to the problem.
Critical weaknesses
- The discussion of computation cost is insufficient. The GPU hours and GPU memory usage should be reported.
- d_p=2304 seems to require a huge computation cost. With the same computational budget, sample-wise similarities could be computed on more samples, which may mitigate the sensitive issue introduced by "mini-batch". Can author provide a comparison or analysis on this?
- The investigation of hyperparameters is not established. What's the effectiveness of \rho, \gamma, \lambda_{DDCM}
Questions for authors
In the first visualization of the distilled data, it shows a man in white shirt, but the caption is "a man in a blue shirt works on a building". What is the insight of this inconsistency on the distilled data?
5 comments
Q2 (Weakness 2): d_p=2304 seems to require a huge computation cost. With the same computational budget, sample-wise similarities could be computed on more samples, which may mitigate the sensitive issue introduced by "mini-batch". Can author provide a comparison or analysis on this?
A2: We thank the reviewer for this suggestion. Although the projection dimension is , DDCM matches only the diagonal of the projected coupling matrix, requiring computation and storage for the matching statistics. FACM computes dense encoder-space couplings with complexity, independent of , whereas computing dense sample-wise similarities requires . Larger real-data batches also increase feature extraction costs through the encoders and projection heads. We therefore compare retrieval performance and total distillation time to examine whether larger sample-wise batches can close the performance gap.
We conduct experiments on Flickr30K with 100 synthetic pairs. For sample-wise matching, we use quantile-based MSE to match the positive- and negative-pair distributions of projected image–text cosine similarities. FACM + Sample-wise combines FACM with this objective instead of DDCM, while Sample-wise uses only this objective. We first compare methods at a fixed real-data batch size of 128, then increase the batch size for both sample-wise variants.
Table R2. Performance and runtime comparison at a fixed real-data batch size of 128 on Flickr30K with 100 synthetic pairs.
| Method | mR | Time | |
|---|---|---|---|
| Sample-wise | 768 | 43.0 | 1.2 h |
| FACM + Sample-wise | 768 | 52.1 | 1.4 h |
| FACM | 768 | 54.5 | 1.0 h |
| FACM + DDCM | 768 | 55.4 | 1.0 h |
| FACM + DDCM | 2304 | 56.3 | 1.0 h |
Table R3. Effect of real-data batch size on sample-wise matching =768.
| Method | Metric | 128 | 192 | 256 | 320 |
|---|---|---|---|---|---|
| Sample-wise | mR | 43.0 | 43.7 | 44.3 | 44.6 |
| Sample-wise | Time | 1.2 h | 1.6 h | 2.0 h | 2.5 h |
| FACM + Sample-wise | mR | 52.1 | 52.8 | 53.0 | 53.3 |
| FACM + Sample-wise | Time | 1.4 h | 2.0 h | 2.7 h | 3.3 h |
At the same projection dimension and real-data batch size , FACM + DDCM achieves 55.4 mR in 1.0 hour, outperforming FACM + Sample-wise by 3.3 points. Increasing the batch size to 320 improves FACM + Sample-wise to 53.3 mR, but also increases its runtime to 3.3 hours. Even with this larger batch, it remains 2.1 points below FACM + DDCM, despite taking 3.3 times as long. Sample-wise alone also improves from 43.0 to 44.6 mR but remains substantially weaker. Increasing from 768 to 2304 further improves FACM + DDCM from 55.4 to 56.3 mR, with both configurations taking approximately 1.0 hour.
These results show that larger sample-wise batches do not close the performance gap with FACM + DDCM within the evaluated range, despite requiring more computation.
We have added this analysis in Appendix D.10 (Table 16), with changes highlighted in blue.
Q3 (Weakness 3): The investigation of hyperparameters is not established. What's the effectiveness of , , ?
A3: We thank the reviewer for this suggestion. We conducted a sensitivity analysis on Flickr30K with 100 synthetic pairs, varying one parameter at a time while keeping the others at their default values $$ρ=1, γ=2, λ_{DDCM}=0.1$). Results are reported in mean Recall (mR$.
DDCM weight . Removing DDCM reduces mR from 56.3 to 54.6 , supporting its complementary contribution to FACM. Performance remains between 55.7 and 56.5 for $\lambda_{DDCM} \in
$, indicating limited sensitivity within this range.
Table R4. Sensitivity to the DDCM weight λ_DDCM.
| 0 | 0.01 | 0.05 | 0.1 | 0.2 | 0.5 | 1 | |
|---|---|---|---|---|---|---|---|
| mR | 54.6 | 55.5 | 55.8 | 56.3 | 56.5 | 55.8 | 55.7 |
FACM exponent . Increasing from 0 (uniform weighting) to 2 improves mR by 6.5 points, demonstrating the benefit of emphasizing more reliable couplings. Performance slightly declines at , suggesting diminishing benefits from stronger weight concentration.
Table R5. Sensitivity to the FACM exponent .
| 0 | 0.5 | 1 | 2 | 4 | |
|---|---|---|---|---|---|
| mR | 49.8 | 51.7 | 55.1 | 56.3 | 55.7 |
Target scaling . The default $\rho=1$ achieves the highest mR , with nearby values of 0.75 and 1.25 yielding 56.0 and 55.8. Reducing to 0.5 lowers mR to 54.1, indicating that substantial downscaling of real-data coupling targets can impair performance.
Table R6. Sensitivity to the target scaling coefficient .
| 0.5 | 0.75 | 1 | 1.25 | 1.5 | |
|---|---|---|---|---|---|
| mR | 54.1 | 56.0 | 56.3 | 55.8 | 55.4 |
We have added the sensitivity analysis in Appendix D.6 Fig\. 8, highlighted in blue.
Q4 (Question 1): In the first visualization of the distilled data, it shows a man in white shirt, but the caption is "a man in a blue shirt works on a building". What is the insight of this inconsistency on the distilled data?
A4: We thank the reviewer for this observation. Following the visualization procedure of CovMatch, we retrieve the nearest caption from the corresponding dataset for each optimized synthetic text embedding. These captions are used only for visualization, while downstream training directly uses the optimized continuous text embeddings.
In the highlighted example, the retrieved caption captures the main activity but differs in shirt color. This illustrates a limitation of using retrieved captions to interpret distilled image–text pairs: they can convey the overall semantics without accurately capturing every visual detail.
We have clarified this point in Appendix E, with changes highlighted in blue.
We appreciate the reviewer's thoughtful comments and address the points below.
Q1 (Weakness 1): The discussion of computation cost is insufficient. The GPU hours and GPU memory usage should be reported.
A1: We thank the reviewer for this comment. We report a computational cost comparison on MS-COCO with 500 synthetic pairs using a single NVIDIA H100 GPU 96GB.
Table R1. Computational cost comparison on MS-COCO with 500 synthetic pairs.
ReCoup completes distillation in 3.5 GPU-hours with 35.9 GB peak GPU memory, compared with 4.8 GPU-hours and 36.6 GB for CovMatch, reducing distillation time by 27.1%. ReCoup also requires no expert preparation or checkpoint storage, unlike LoRS and RepBlend, which incur 136.1 and 77.8 hours of additional expert preparation, respectively.
These results show that ReCoup achieves its retrieval improvements with lower distillation time and comparable peak GPU memory relative to CovMatch.
We have revised Section 4.4 (Table 2), with changes highlighted in blue.