- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Progressive Linearized Trajectory Matching (PLTM) for distilling image–text datasets. It replaces epoch-dependent expert targets with intervals along a straight line connecting selected expert endpoints, progressively increasing the interval during distillation. Under hard supervision, it adds alignment between image and text Gram matrices and an intra-modal diversity penalty. Experiments demonstrate retrieval improvements on Flickr-30K and MS-COCO, with additional architecture-transfer and asymmetric-architecture evaluations.
Critical strengths
-
The method is simple and practically attractive. Endpoint interpolation and a scalar interval schedule require little additional machinery. The regularizers reuse existing representations and avoid introducing an optimized similarity matrix in the hard-supervision setting. This simplicity is valuable if the improvements remain robust under controlled comparisons.
-
On Flickr-30K with 500 pairs, IR@10 increases from 37.7 to 56.3 and TR@10 from 57.1 to 70.0 over RepBlend without soft similarities. MS-COCO also improves, from 23.1 to 30.9 and from 32.6 to 38.7, respectively. These are meaningful gains within the evaluated protocol.
Critical weaknesses
-
The motivation does not establish the proposed optimization mechanism. Figure 1 demonstrates changing epoch-update directions and repeatable within-expert angular patterns. However, the low variance of these patterns across experts does not establish that straightening trajectories makes synthetic-data optimization easier.
-
The ablations do not verify the effectiveness of the progressive schedule or the regularizers. Table 3 compares progression against one fixed long interval, on one dataset and budget. It omits fixed short/intermediate intervals, randomly ordered interval lengths, and a reversed schedule. These controls matter because changing the interval also changes the initialization distribution and the normalization denominator in Equation 9. The regularizers should additionally be tested on the original trajectory-matching baseline to determine whether their benefits depend on PLTM.
-
Some baselines are not missing. This paper does not report the results from competitive baselines, such as CovMatch [1] and APM [2].
-
The approximately twofold efficiency claim lacks direct measurement. Using 1,000 outer iterations instead of 2,000 establishes a reduction in iteration count. However, this paper does not establish a corresponding wall-clock speedup or negligible overhead. It provides no runtime table, memory measurements, or retrieval-versus-time curves.
[1] CovMatch: Cross-covariance guided multimodal dataset distillation with trainable text encoder. NeurIPS 2026.
[2] Efficient Multi-modal Dataset Distillation via Analytic Parameter Matching. ICML 2026.
Questions for authors
See weaknesses.
4 comments
W2. Stronger controls for the schedule and regularizers
A2. The additional controls support two conclusions: short-to-long progression outperforms fixed, reversed, and random schedules, and the combined regularization benefit does not depend on PLTM. The evidence is as follows:
(1) A fixed short interval does not reproduce the benefit of progression. Short-to-long achieves the highest mR, 44.07 (Table 4).
Table 4. Matching-interval schedule controls: Flickr-30K, 500 pairs.
| Schedule | mR | Gain of short-to-long over this control (%) |
|---|---|---|
| Progressive short-to-long | 44.07 | — |
| Fixed short interval | 42.55 | 1.52 |
| Fixed intermediate interval | 40.15 | 3.92 |
| Reversed long-to-short | 42.39 | 1.68 |
| Random interval length sampled at each iteration | 42.31 | 1.76 |
Short-to-long progression improves over fixed short, reversed, and random schedules by 1.52%, 1.68%, and 1.76%, respectively. Table 3 of the paper also shows 40.9 for fixed long intervals versus 43.3 for progression.
With no student update, each modality’s normalized error is 1 for any nonzero interval. Progression increases target displacement as synthetic data improve; it also changes the starting-point sampling range and normalization denominator.
(2) Both regularizers remain effective on the original-trajectory baseline. Table 5 adds both terms to RepBlend w/o Sim.
Table 5. Adding both regularizers to RepBlend w/o Sim with the original trajectory.
| Flickr-30K synthetic pairs | Original-trajectory baseline | With relational alignment and diversity regularization | Gain (%) |
|---|---|---|---|
| 100 | 30.03 | 33.02 | 2.99 |
| 200 | 30.02 | 35.12 | 5.1 |
| 500 | 31.7 | 35.57 | 3.87 |
As shown in Table 5, gains of 2.99%, 5.10%, and 3.87% across the three budgets show that the combined regularization benefit does not depend on PLTM. Table 3 of the paper further shows that adding relational alignment to diversity regularization yields +0.8%, supporting their complementarity.
W3. Comparisons with CovMatch and APM
A3. The CovMatch comparison is already included in Appendix D of the paper. APM uses a different analytic matching setup; we will cite this work and design a fair comparison under common conditions:
(1) The existing CovMatch comparison shows higher mR for our method on both datasets. Tables 10–11 in Appendix D of the paper show gains of 4.9% on Flickr-30K and 3.1% on MS-COCO.
We will highlight this comparison and specify matched modules, trainable parameters, and protocols.
(2) APM provides a valuable analytic parameter matching approach; a fair comparison requires a common matching setup. APM analytically matches linear projections; we address intermediate-state dependence and target magnitudes, also considering asymmetric image-encoder/text-projection matching. Different matched modules confound direct comparison. We will add appendix results under comparable conditions.
W4. Is the approximately twofold efficiency improvement supported by timing data?
A4. Yes, estimates based on measured per-iteration costs show that PLTM roughly halves the distillation-loop runtime while achieving better retrieval performance:
(1) Similar per-iteration costs and half as many iterations yield nearly a twofold speedup. Table 6 gives measured per-iteration costs and estimated loop runtimes: 1.96× and 1.99× speedups.
Table 6. Measured per-iteration costs and distillation-loop runtimes estimated from iteration budgets.
| Dataset | RepBlend per iteration | PLTM per iteration | Per-iteration overhead | RepBlend: 2,000 iterations | PLTM: 1,000 iterations | Speedup |
|---|---|---|---|---|---|---|
| Flickr-30K | 1.46 s | 1.49 s | 2.10% | 2,920 s (48.7 min) | 1,490 s (24.8 min) | 1.96× |
| MS-COCO | 3.36 s | 3.38 s | 0.60% | 6,720 s (112.0 min) | 3,380 s (56.3 min) | 1.99× |
PLTM adds only 0.6%–2.1% per-iteration overhead while reducing the iteration count from 2,000 to 1,000, yielding approximately 49% less distillation-loop runtime.
(2) Time savings accompany better retrieval quality. Table 3 shows 44.39 mR in 1,000 PLTM iterations versus 42.63 in 2,000 RepBlend iterations (w/ Sim).
Dear Reviewer mNir,
Thank you again for your valuable comments on our submission. We would like to kindly confirm whether we have sufficiently addressed all of your concerns (or at least part of them).
Should there be any remaining questions or areas requiring further clarification, please do not hesitate to let us know.
If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.
We sincerely look forward to your feedback.
Thank you for your constructive suggestions on empirical validation and computational efficiency.
W1. Do direction statistics establish that linearization improves optimization?
A1. Reducing dependence on intermediate expert states and progressively increasing the target magnitude as the synthetic data are optimized improves final retrieval performance and reduces outer iterations. Figure 1 identifies directional changes; the target ablation in Table 3 of the paper and iteration results in Table 3 evaluate the practical benefits:
(1) Matching requirements can remain tied to intermediate expert states after direction smoothing. Synthetic pairs must produce both the required direction and magnitude. Expert-defined magnitudes do not adjust as the pairs improve. Within each window, PLTM uses the start-to-end update Δ and an increasing interval wₖ, giving magnitude wₖ‖Δ‖ without intermediate states.
(2) Ablations evaluate the benefits of removing intermediate-state dependence and adjusting the target magnitude. Table 3 of the paper shows 31.8 → 40.9 mR from original targets to LTM, then +2.4% from progression: both removing intermediate-state dependence and increasing target magnitude contribute.
(3) Path diagnostics further distinguish the expert's intermediate training process from its value for distillation. For 20 experts over epochs 0–4, the joint image–text parameter path is 1.301 times the endpoint distance, with 22.7% mean deviation from the chord (Table 1). Table 2 compares validation loss.
Table 1. Path geometry within the actual matching window: 20 experts, epochs 0–4.
Table 2. Validation loss along the original expert trajectory and endpoint chord: symmetric InfoNCE on the same 512 pairs.
The chord has higher real-data loss at these intermediate positions, yet data distilled using it as the target achieve better downstream retrieval. Together with Table 3 of the paper, this distinguishes intermediate-model quality from distillation utility.
Table 3. Retrieval performance at different outer iterations: Flickr-30K, 500 pairs; entries are mR.
Table 3 shows 44.39 mR in 1,000 iterations for PLTM versus 42.63 in 2,000 for RepBlend (w/ Sim); PLTM w/o Sim reaches 44.29 in a 450-iteration run. Learning the overall update with progressively increasing displacement improves retrieval and iteration efficiency.