acceptodds
Under review as a conference paper at ICLR 2027

Not Every Turn Matters: Progressive Linearized Trajectory Matching for Multimodal Dataset Distillation

Abstract

Multimodal Dataset Distillation (MDD) compresses image–text datasets into synthetic pairs that retain cross-modal retrieval utility. Existing trajectory-based methods use one-epoch expert updates as matching targets. However, each target direction is tied to a sampled intermediate epoch, so a severely compressed synthetic set must reproduce directions that change frequently across training. Across 20 independently trained experts, we observe substantial direction changes across epochs but a consistent cross-epoch trend among experts. We therefore propose Progressive Linearized Trajectory Matching (PLTM), which aligns every target with the overall direction from the selected start state to the end state and progressively increases the matched interval as the synthetic pairs improve. This redesign provides a direction-consistent expert target. Under paired hard supervision, online relational alignment and diversity regularization preserve relations among pairs and sample diversity without optimizing an additional soft similarity matrix. PLTM incurs negligible computational overhead and approximately doubles distillation efficiency. Experiments on Flickr-30K and MS-COCO show that our method consistently outperforms previous state-of-the-art MDD methods under paired hard supervision and optional soft similarity supervision. On MS-COCO with 500 synthetic pairs, PLTM improves IR@10/TR@10 by 7.8%/6.1% without a soft similarity matrix and by 4.2%/4.4% with soft supervision.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Dataset Distillation, Representation Learning, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mNirOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Progressive Linearized Trajectory Matching (PLTM) for distilling image–text datasets. It replaces epoch-dependent expert targets with intervals along a straight line connecting selected expert endpoints, progressively increasing the interval during distillation. Under hard supervision, it adds alignment between image and text Gram matrices and an intra-modal diversity penalty. Experiments demonstrate retrieval improvements on Flickr-30K and MS-COCO, with additional architecture-transfer and asymmetric-architecture evaluations.

Critical strengths
  1. The method is simple and practically attractive. Endpoint interpolation and a scalar interval schedule require little additional machinery. The regularizers reuse existing representations and avoid introducing an optimized similarity matrix in the hard-supervision setting. This simplicity is valuable if the improvements remain robust under controlled comparisons.

  2. On Flickr-30K with 500 pairs, IR@10 increases from 37.7 to 56.3 and TR@10 from 57.1 to 70.0 over RepBlend without soft similarities. MS-COCO also improves, from 23.1 to 30.9 and from 32.6 to 38.7, respectively. These are meaningful gains within the evaluated protocol.

Critical weaknesses
  1. The motivation does not establish the proposed optimization mechanism. Figure 1 demonstrates changing epoch-update directions and repeatable within-expert angular patterns. However, the low variance of these patterns across experts does not establish that straightening trajectories makes synthetic-data optimization easier.

  2. The ablations do not verify the effectiveness of the progressive schedule or the regularizers. Table 3 compares progression against one fixed long interval, on one dataset and budget. It omits fixed short/intermediate intervals, randomly ordered interval lengths, and a reversed schedule. These controls matter because changing the interval also changes the initialization distribution and the normalization denominator in Equation 9. The regularizers should additionally be tested on the original trajectory-matching baseline to determine whether their benefits depend on PLTM.

  3. Some baselines are not missing. This paper does not report the results from competitive baselines, such as CovMatch [1] and APM [2].

  4. The approximately twofold efficiency claim lacks direct measurement. Using 1,000 outer iterations instead of 2,000 establishes a reduction in iteration count. However, this paper does not establish a corresponding wall-clock speedup or negligible overhead. It provides no runtime table, memory measurements, or retrieval-versus-time curves.

[1] CovMatch: Cross-covariance guided multimodal dataset distillation with trainable text encoder. NeurIPS 2026.

[2] Efficient Multi-modal Dataset Distillation via Analytic Parameter Matching. ICML 2026.

Questions for authors

See weaknesses.

4 comments

Thank you for your constructive suggestions on empirical validation and computational efficiency.

W1. Do direction statistics establish that linearization improves optimization?

A1. Reducing dependence on intermediate expert states and progressively increasing the target magnitude as the synthetic data are optimized improves final retrieval performance and reduces outer iterations. Figure 1 identifies directional changes; the target ablation in Table 3 of the paper and iteration results in Table 3 evaluate the practical benefits:

(1) Matching requirements can remain tied to intermediate expert states after direction smoothing. Synthetic pairs must produce both the required direction and magnitude. Expert-defined magnitudes do not adjust as the pairs improve. Within each window, PLTM uses the start-to-end update Δ and an increasing interval wₖ, giving magnitude wₖ‖Δ‖ without intermediate states.

(2) Ablations evaluate the benefits of removing intermediate-state dependence and adjusting the target magnitude. Table 3 of the paper shows 31.8 → 40.9 mR from original targets to LTM, then +2.4% from progression: both removing intermediate-state dependence and increasing target magnitude contribute.

(3) Path diagnostics further distinguish the expert's intermediate training process from its value for distillation. For 20 experts over epochs 0–4, the joint image–text parameter path is 1.301 times the endpoint distance, with 22.7% mean deviation from the chord (Table 1). Table 2 compares validation loss.

Table 1. Path geometry within the actual matching window: 20 experts, epochs 0–4.

DiagnosticResultInterpretation
Recorded path length / endpoint chord length1.301The path is, on average, 30.1% longer than the endpoint chord
Mean intermediate-checkpoint deviation from the chord / endpoint distance22.70%Intermediate states exhibit measurable geometric deviation

Table 2. Validation loss along the original expert trajectory and endpoint chord: symmetric InfoNCE on the same 512 pairs.

Progress through the windowOriginal expert trajectoryEndpoint chord
25%2.1943.96
50%2.0062.688
75%1.9172.01

The chord has higher real-data loss at these intermediate positions, yet data distilled using it as the target achieve better downstream retrieval. Together with Table 3 of the paper, this distinguishes intermediate-model quality from distillation utility.

Table 3. Retrieval performance at different outer iterations: Flickr-30K, 500 pairs; entries are mR.

Outer iterationsMCT, w/o SimRepBlend, w/ SimPLTM, w/ Sim (total budget 1,000)PLTM, w/o Sim (total budget 600)PLTM, w/o Sim (total budget 450)
15041.4113.9940.4341.4640.35
30042.7721.3339.7642.5242.25
45041.2128.6741.0944.2144.29
60037.9934.4241.8944.09—
75036.1235.2942.42——
90035.5535.4243.83——
1,000—37.3944.39——
2,00032.5542.63———

Table 3 shows 44.39 mR in 1,000 iterations for PLTM versus 42.63 in 2,000 for RepBlend (w/ Sim); PLTM w/o Sim reaches 44.29 in a 450-iteration run. Learning the overall update with progressively increasing displacement improves retrieval and iteration efficiency.

W2. Stronger controls for the schedule and regularizers

A2. The additional controls support two conclusions: short-to-long progression outperforms fixed, reversed, and random schedules, and the combined regularization benefit does not depend on PLTM. The evidence is as follows:

(1) A fixed short interval does not reproduce the benefit of progression. Short-to-long achieves the highest mR, 44.07 (Table 4).

Table 4. Matching-interval schedule controls: Flickr-30K, 500 pairs.

SchedulemRGain of short-to-long over this control (%)
Progressive short-to-long44.07—
Fixed short interval42.551.52
Fixed intermediate interval40.153.92
Reversed long-to-short42.391.68
Random interval length sampled at each iteration42.311.76

Short-to-long progression improves over fixed short, reversed, and random schedules by 1.52%, 1.68%, and 1.76%, respectively. Table 3 of the paper also shows 40.9 for fixed long intervals versus 43.3 for progression.

With no student update, each modality’s normalized error is 1 for any nonzero interval. Progression increases target displacement as synthetic data improve; it also changes the starting-point sampling range and normalization denominator.

(2) Both regularizers remain effective on the original-trajectory baseline. Table 5 adds both terms to RepBlend w/o Sim.

Table 5. Adding both regularizers to RepBlend w/o Sim with the original trajectory.

Flickr-30K synthetic pairsOriginal-trajectory baselineWith relational alignment and diversity regularizationGain (%)
10030.0333.022.99
20030.0235.125.1
50031.735.573.87

As shown in Table 5, gains of 2.99%, 5.10%, and 3.87% across the three budgets show that the combined regularization benefit does not depend on PLTM. Table 3 of the paper further shows that adding relational alignment to diversity regularization yields +0.8%, supporting their complementarity.

W3. Comparisons with CovMatch and APM

A3. The CovMatch comparison is already included in Appendix D of the paper. APM uses a different analytic matching setup; we will cite this work and design a fair comparison under common conditions:

(1) The existing CovMatch comparison shows higher mR for our method on both datasets. Tables 10–11 in Appendix D of the paper show gains of 4.9% on Flickr-30K and 3.1% on MS-COCO.

We will highlight this comparison and specify matched modules, trainable parameters, and protocols.

(2) APM provides a valuable analytic parameter matching approach; a fair comparison requires a common matching setup. APM analytically matches linear projections; we address intermediate-state dependence and target magnitudes, also considering asymmetric image-encoder/text-projection matching. Different matched modules confound direct comparison. We will add appendix results under comparable conditions.

W4. Is the approximately twofold efficiency improvement supported by timing data?

A4. Yes, estimates based on measured per-iteration costs show that PLTM roughly halves the distillation-loop runtime while achieving better retrieval performance:

(1) Similar per-iteration costs and half as many iterations yield nearly a twofold speedup. Table 6 gives measured per-iteration costs and estimated loop runtimes: 1.96× and 1.99× speedups.

Table 6. Measured per-iteration costs and distillation-loop runtimes estimated from iteration budgets.

DatasetRepBlend per iterationPLTM per iterationPer-iteration overheadRepBlend: 2,000 iterationsPLTM: 1,000 iterationsSpeedup
Flickr-30K1.46 s1.49 s2.10%2,920 s (48.7 min)1,490 s (24.8 min)1.96×
MS-COCO3.36 s3.38 s0.60%6,720 s (112.0 min)3,380 s (56.3 min)1.99×

PLTM adds only 0.6%–2.1% per-iteration overhead while reducing the iteration count from 2,000 to 1,000, yielding approximately 49% less distillation-loop runtime.

(2) Time savings accompany better retrieval quality. Table 3 shows 44.39 mR in 1,000 PLTM iterations versus 42.63 in 2,000 RepBlend iterations (w/ Sim).

Dear Reviewer mNir,

Thank you again for your valuable comments on our submission. We would like to kindly confirm whether we have sufficiently addressed all of your concerns (or at least part of them).

Should there be any remaining questions or areas requiring further clarification, please do not hesitate to let us know.

If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Reviewer_ZcPqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Progressive Linearized Trajectory Matching (PLTM) for multimodal dataset distillation. PLTM replaces the original expert trajectory with a straight path between its start and end states and gradually increases the matching interval during distillation. The authors also introduce relational alignment and diversity regularization to improve performance under hard supervision without learning an additional soft similarity matrix. Experiments on Flickr-30K and MS-COCO show competitive retrieval performance.

Critical strengths
  1. The proposed method is simple and easy to integrate into existing trajectory-matching frameworks, with little additional computational overhead.
  2. The paper is clearly written and easy to follow. The experiments show promising improvements, especially under hard supervision. The authors also evaluate different data budgets and model architectures.
Critical weaknesses
  1. Prior works such as MCT (Zhong et al., CVPR 2025) and PTM-ST (Guo et al., ICLR 2026) have already discussed the instability of expert trajectories and explored interpolation-based strategies to construct smoother matching trajectories. Table 3 shows that most of the performance gain comes from linearization, while progression provides a smaller improvement. This raises concerns about the novelty of the main contribution, as both the motivation and the core method are closely related to prior work.
  2. More controlled comparisons with existing interpolation strategies are needed. Although Appendix D compares PLTM with PTM-ST, their main experiments match different model components. PLTM matches image and text projection modules, while PTM-ST matches the image encoder and text projection module. Thus, the performance differences may partly come from the different optimization settings rather than the proposed trajectory matching strategy. A controlled comparison using PTM-ST's Shortcut Trajectory under the same framework would better clarify the actual contribution of PLTM.
  3. The authors argue that relational alignment compensates for the lack of fine-grained supervision under hard labels. However, aligning image–image and text–text similarity matrices does not necessarily preserve meaningful cross-modal relationships. Moreover, related work such as CovMatch has already explored preserving multimodal relationships through feature statistics, although with a different formulation. Table 3 also shows that relational alignment alone provides only a 0.2-point improvement, while most of the gains under hard supervision come from trajectory matching.
Questions for authors
  1. What are the key technical differences between PLTM's trajectory linearization and existing interpolation-based methods such as MCT and PTM-ST?

  2. Could the authors replace PLTM's linearization strategy with PTM-ST's Shortcut Trajectory under the same framework to better demonstrate the independent benefits of the proposed method?

  3. How does aligning image–image and text–text similarities compensate for the missing fine-grained cross-modal supervision? Given its limited improvement in Table 3, could the authors provide further evidence of the effectiveness of relational alignment?

  4. Why are shorter matching intervals assumed to be easier to optimize early in distillation? The proposed schedule depends only on the number of distillation iterations rather than the actual matching difficulty. Moreover, since the matching loss is normalized by the squared target displacement, shorter intervals are not necessarily easier to match. Could the authors provide further justification or evidence for this assumption?

4 comments

Thank you for your careful assessment of our method and suggestions for fairer comparisons.

W1 & Q1. How does the research question differ from MCT/PTM-ST?

A1. Yes, MCT/PTM-ST have improved trajectory stability, but they still depend on intermediate expert states. This leaves two key issues:

(1) Even after trajectory smoothing, the synthetic data must still adapt to stage-dependent updates. The limited pairs must learn the overall training effect while fitting local updates defined by intermediate states.

(2) Matching is difficult for a small synthetic set because both update directions and magnitudes vary. Expert-defined magnitudes still fluctuate and do not adjust as synthetic data improve. MCT declines after 300 outer iterations (Table 2). A separate EMA experiment also shows a decline. PLTM w/ Sim improves across the reported 300–1,000 checkpoints.

Our contribution is therefore to improve directional consistency while removing the dependence on intermediate expert states for determining the target update magnitude. Within each expert window, the overall start-to-end update Δ sets the direction, and the progressively increasing matching interval wₖ sets the magnitude wₖ‖Δ‖. Targets no longer fluctuate with the sampled intermediate position; their magnitude grows with distillation progress, reducing the matching burden.

Yes, linearization accounts for the main 9.1% gain. This directly answers our central question: a small synthetic set can train fresh students more effectively without satisfying all of the expert's stage-dependent matching requirements. Progression adds 2.4%. Despite lower intermediate loss on the original trajectory (Table 1), linearized targets yield better distilled data (Table 3 of the paper): intermediate model quality and distillation utility differ.

Table 1. Validation loss along the original expert trajectory and endpoint chord: symmetric InfoNCE on the same 512 pairs.

Progress through the windowOriginal expert trajectoryEndpoint chord
25%2.1943.96
50%2.0062.688
75%1.9172.01

Table 2. Retrieval performance at different outer iterations: Flickr-30K, 500 pairs; entries are mR.

Outer iterationsMCT, w/o SimRepBlend, w/ SimPLTM, w/ Sim (total budget 1,000)PLTM, w/o Sim (total budget 600)PLTM, w/o Sim (total budget 450)
15041.4113.9940.4341.4640.35
30042.7721.3339.7642.5242.25
45041.2128.6741.0944.2144.29
60037.9934.4241.8944.09—
75036.1235.2942.42——
90035.5535.4243.83——
1,000—37.3944.39——
2,00032.5542.63———

With w/ Sim, PLTM reaches 44.39 mR in 1,000 iterations versus RepBlend’s 42.63 in 2,000, at similar per-iteration cost (Table 2).

MCT peaks at 42.77 after 300 iterations, then falls to 32.55 at 2,000. PLTM w/o Sim reaches 44.29 in a 450-iteration run (+1.52%). Thus, effective optimization remains an issue after smoothing; the controlled PTM-ST comparison below further tests our target choice.

W2 & Q2. Controlled Shortcut Trajectory replacement under the same experimental setup

A2. Yes, we completed a controlled Shortcut Trajectory replacement experiment under the same setup. PLTM still improves retrieval performance when the matched modules are held fixed:

(1) We hold the matched modules fixed and compare trajectory targets. We replace Shortcut Trajectory with PLTM while retaining the same projection-matching setup (Table 3).

Table 3. PTM-ST versus PTM-ST + PLTM under the same projection-matching setup.

MetricPTM-STPTM-ST + PLTM
IR@115.72 ± 0.2615.69 ± 0.05
IR@540.06 ± 0.2640.76 ± 0.63
IR@1052.46 ± 0.2653.86 ± 0.68
TR@122.26 ± 0.6322.56 ± 0.85
TR@551.58 ± 0.5252.50 ± 0.23
TR@1066.42 ± 0.7066.36 ± 0.45
mR41.42 ± 0.3441.95 ± 0.29

(2) The benefit is reflected in overall retrieval performance under the same setup. Table 3 shows +0.53% mR (41.42 ± 0.34 → 41.95 ± 0.29), +1.40% IR@10, and +0.92% TR@5.

W3 & Q3. How does within-modality relational alignment help cross-modal supervision?

A3. Relational alignment supplements relations among samples that hard labels do not specify, and complements diversity regularization. These regularizers are auxiliary components for paired hard supervision, with their roles and benefits explained below:

(1) Shared pair indices turn within-modality relations into a cross-modal constraint.

Hard labels identify image–text pairs without specifying fine-grained relations between different pairs. Relational alignment adds: if images i and j are similar in the current representation, the relationship between their corresponding texts i and j should be consistent with that similarity. Shared pair indices connect the two modalities.

(2) Relational alignment and diversity regularization are more effective together. Consistent relations can still be redundant; diversity regularization addresses this (Table 3 of the paper).

The gain from relational alignment alone is indeed small; with diversity regularization already present, adding relational alignment yields a 0.8% improvement (44.0 → 44.8).

(3) The combined regularization benefit does not depend on PLTM. Table 4 adds both terms to RepBlend w/o Sim with the original trajectory.

Table 4. Adding both regularizers to RepBlend w/o Sim with the original trajectory.

Flickr-30K synthetic pairsOriginal baselineWith both regularizersGain (%)
10030.0333.022.99
20030.0235.125.1
50031.735.573.87

CovMatch matches real–synthetic cross-covariance and related statistics. Our terms constrain relations and diversity within the synthetic pairs, improving training utility under paired hard supervision without an additional learnable soft similarity matrix.

Q4. Are shorter intervals necessarily easier, and why use an iteration-based progressive schedule?

A4. Shorter intervals do not necessarily have lower normalized loss. We use progression to increase the target displacement as the synthetic data are optimized, and experiments show that this is more effective than fixed, reversed, and random schedules:

(1) The schedule adjusts target displacement; it does not assume a lower normalized loss. The interval increases with outer iterations as the synthetic pairs are optimized.

The target displacement is wΔ; Equation (9) normalizes by w²‖Δ‖². With no student update, each modality’s normalized error is 1 for any nonzero w. Shorter displacement therefore need not mean lower loss. Changing w also changes the starting-point sampling range and loss scale.

(2) Controlled experiments support progressively increasing the target magnitude. Short-to-long achieves 44.07 mR (Table 5).

Table 5. Matching-interval schedule controls: Flickr-30K, 500 pairs.

SchedulemRGain of short-to-long over this control (%)
Progressive short-to-long44.07—
Fixed short interval42.551.52
Fixed intermediate interval40.153.92
Reversed long-to-short42.391.68
Random interval length sampled at each iteration42.311.76

Short-to-long progression improves over the fixed short interval by 1.52%, and over reversed and random schedules by 1.68% and 1.76%, respectively. We will describe the motivation as progressively increasing the matching requirement.

Dear Reviewer ZcPq,

Thank you again for your valuable comments on our submission. We would like to kindly confirm whether we have sufficiently addressed all of your concerns (or at least part of them).

Should there be any remaining questions or areas requiring further clarification, please do not hesitate to let us know.

If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Reviewer_uiTkOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PLTM for image–text dataset distillation. From 20 independently trained experts, it observes that one-epoch parameter updates change direction substantially across epochs, while the pattern is consistent across experts. PLTM replaces epoch-wise targets with segments of the chord between the first and last expert states and lengthens the matched segment over outer iterations. For hard supervision it adds a Gram-matrix alignment loss and a diversity penalty.

Critical strengths
  1. The observation is concrete and the resulting change is simple. Figure 1 quantifies directional variation across experts, and the method only needs start and end states per expert. It works with both hard and soft supervision and can be stacked on LoRS.
  2. The hard-label results question whether a learnable similarity matrix is necessary.
  3. The evaluation spans several important axes. Two supervision regimes, symmetric and asymmetric trajectories, two unseen architectures, and a component ablation, with a stated fair-comparison protocol restricting all methods to the same early-trajectory window.
Critical weaknesses
  1. The core idea already exists and is not compared. PTM-ST (Guo et al., ICLR 2026) builds a shortcut trajectory θt=(1−β(t))θ0+β(t)θtpθ_t = (1−β(t))θ_0 + β(t)θ_{t_p} for image–text distillation, motivated identically by low directional consistency. MCT (Zhong et al., CVPR 2025) matches a convexified start-to-end trajectory in the unimodal case. Neither appears in the baselines; the introduction presents linearization as novel. The window schedule echoes DATM/ATT; the regularizers are relatives of RKD, similarity-preserving KD, alignment/uniformity, VICReg, and CovMatch. What PLTM specifically adds—a single chord with uniform sub-segments and a monotone window—needs stating, and matched-protocol comparisons with PTM-ST and MCT inside the same RepBlend pipeline are essential.

  2. The theory does not establish why the chord transfers better. MTT-style methods already match only the endpoint of one M-epoch segment; no target requires reproducing every intermediate update, so the framing somewhat misdescribes the prior work. Proposition 1 shows fewer conditions, not that interpolated initializations are meaningful or that the chord preserves what a fresh student needs. Low epoch-wise cosine similarity can also arise from SGD noise while the cumulative path stays nearly straight. I would like to see path straightness, cos(Δ0:t,Δ0:R)cos(Δ_{0:t}, Δ_{0:R}), and real-data loss along the chord versus the true trajectory within the actual window used.

References:

  1. Guo, S., Zhao, H., Yang, S., Jiang, C., Cheng, Y., Peng, X., Shao, R., & Tian, Z. (2026). Multimodal dataset distillation via phased teacher models (PTM-ST). ICLR 2026.
  2. Zhong, W., Tang, H., Zheng, Q., Xu, M., Hu, Y., & Guan, W. (2025). Towards stable and storage-efficient dataset distillation: Matching convexified trajectory (MCT). CVPR 2025
Questions for authors

APM (Bo & Wang, 2026, ICML 2026), which reports IR@1 = 12.8 and TR@1 = 17.8 on Flickr-30K with 100 pairs in a projector-based formulation (protocols may differ; PLTM's best in Table 1 is 12.5 and 17.4), a discussion or comparison would help position the state-of-the-art claim. In line with ICLR's policy on contemporaneous work, I do not weigh its absence in my score.

4 comments

Thank you for your suggestions.

W1. What problem does PLTM address beyond existing interpolation methods?

A1. Yes, MCT/PTM-ST have improved trajectory stability, but they still depend on intermediate expert states. This leaves two key issues:

(1) Even after trajectory smoothing, the synthetic data must still adapt to stage-dependent updates. The limited pairs must learn the overall training effect while fitting local updates defined by intermediate states.

(2) Matching is difficult for a small synthetic set because both update directions and magnitudes vary. Expert-defined magnitudes still fluctuate and do not adjust as synthetic data improve. MCT declines after 300 outer iterations (Table 1). A separate EMA experiment also shows a decline. PLTM w/ Sim improves across the reported 300–1,000 checkpoints.

Our contribution is therefore to improve directional consistency while removing the dependence on intermediate expert states for determining the target update magnitude. Within each expert window, the overall start-to-end update Δ sets the direction, and the progressively increasing matching interval wₖ sets the magnitude wₖ‖Δ‖. Targets no longer fluctuate with the sampled intermediate position; their magnitude grows with distillation progress, reducing the matching burden.

LTM raises mR from 31.8 to 40.9, and progressive matching further raises it to 43.3. Table 3 of the paper supports removing stage-dependent requirements; the additional 2.4% supports progressively increasing the matching magnitude. Lower intermediate expert loss does not imply better distilled data (W2).

Table 1. Retrieval performance at different outer iterations: Flickr-30K, 500 pairs; entries are mR.

Outer iterationsMCT, w/o SimRepBlend, w/ SimPLTM, w/ Sim (total budget 1,000)PLTM, w/o Sim (total budget 600)PLTM, w/o Sim (total budget 450)
15041.4113.9940.4341.4640.35
30042.7721.3339.7642.5242.25
45041.2128.6741.0944.2144.29
60037.9934.4241.8944.09—
75036.1235.2942.42——
90035.5535.4243.83——
1,000—37.3944.39——
2,00032.5542.63———

As shown in Table 1, trajectory smoothing alone does not ensure continued improvement in retrieval during distillation. MCT peaks at 42.77 after 300 iterations and falls to 32.55 at 2,000. PLTM w/o Sim reaches 44.29 in a 450-iteration run (+1.52% over that peak); PLTM w/ Sim rises from 39.76 to 44.39 across the reported 300–1,000 checkpoints.

This adjustment improves both quality and efficiency. With w/ Sim, PLTM reaches 44.39 mR in 1,000 iterations versus RepBlend’s 42.63 in 2,000, at similar per-iteration cost.

Replacing Shortcut Trajectory with PLTM under identical matched modules raises mR from 41.42 ± 0.34 to 41.95 ± 0.29 (+0.53%).

We will acknowledge DATM/ATT’s scheduling foundations. Regarding regularization:

(1) The regularizers are auxiliary components for paired hard supervision, rather than our core contribution. Relational alignment supplements relations unspecified by hard labels; diversity regularization reduces redundancy, without an additional learnable soft similarity matrix.

(2) We build on relational preservation and prevention of representation collapse, but apply these ideas to different objects and objectives. RKD/similarity-preserving KD transfer teacher–student sample relations; alignment/uniformity concerns contrastive alignment and representation distributions; VICReg prevents self-supervised collapse; CovMatch matches real–synthetic statistics. Here, relational alignment matches image/text Gram matrices with shared pair indices, while diversity regularization reduces within-modality redundancy under paired hard supervision. We will clarify these connections.

W2. Is the chord meaningful, and does the theory establish better transfer?

A2. The expert path within the actual matching window is not nearly straight. The endpoint chord is useful because it yields synthetic data with better retrieval performance, rather than intermediate models with lower validation loss. Geometric diagnostics and retrieval experiments support these two observations:

(1) The cumulative path exhibits clear deviations; directional changes are not confined to local updates.

Table 2. Path geometry within the actual matching window: 20 experts, epochs 0–4.

DiagnosticResultInterpretation
Recorded path length / endpoint chord length1.301The path is, on average, 30.1% longer than the endpoint chord
Mean intermediate-checkpoint deviation from the chord / endpoint distance22.70%Intermediate states exhibit measurable geometric deviation

(2) Lower loss at intermediate expert states does not imply more effective distilled data. Table 3 compares symmetric InfoNCE on the same 512 validation pairs.

Table 3. Validation loss along the original expert trajectory and endpoint chord: symmetric InfoNCE on the same 512 pairs.

Progress through the windowOriginal expert trajectoryEndpoint chord
25%2.1943.96
50%2.0062.688
75%1.9172.01

The original trajectory has lower intermediate loss, whereas the linearized target yields synthetic data with better retrieval performance. PLTM targets the utility of data for training fresh students, supported by the 31.8 → 40.9 → 43.3 mR results in Table 3 of the paper.

(3) Proposition 1 explains the matching constraints; retrieval experiments evaluate synthetic-data utility. You are correct: MTT matches sampled interval endpoints. We will clarify that their update directions and magnitudes vary across intervals. Proposition 1 establishes that epoch-wise matching is sufficient, not necessary, for matching the overall start-to-end update; Table 3 of the paper tests retrieval utility.

Q1. Comparison with APM

A3. APM provides a valuable analytic parameter matching approach to multimodal dataset distillation, and we will cite and discuss it in the revision. The main differences concern the research objective and matching setup:

(1) The research approaches differ. APM uses analytic matching of linear projections; we address intermediate-state dependence and target magnitudes in trajectory matching.

(2) The matching setups differ, requiring common conditions for a fair comparison. APM assumes linear projection layers; we also match an image encoder and text projection asymmetrically. A direct comparison would confound method and module differences. We will clarify the settings and add appendix results.

Dear Reviewer uiTK,

Thank you again for your valuable comments on our submission. We would like to kindly confirm whether we have sufficiently addressed all of your concerns (or at least part of them).

Should there be any remaining questions or areas requiring further clarification, please do not hesitate to let us know.

If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Comments on the paper

2 comments

Dear Reviewers,

As the discussion deadline (November 18, AoE) approaches, please review the authors’ rebuttal if you have not already done so and post a follow-up confirming that you have read it and indicating whether your main concerns have been addressed.

Best, AC

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Revised Manuscript

Dear Reviewer,

Thank you again for your thoughtful feedback. Following your suggestions, we have provided detailed responses and revised the manuscript, with additional experiments presented primarily in Appendix F. The updates include:

  • Clarifying PLTM’s contribution beyond trajectory smoothing: removing dependence on intermediate expert states while progressively increasing the target update magnitude.
  • Adding path-geometry and validation-loss diagnostics to distinguish intermediate-model quality from distilled-data utility.
  • Reporting retrieval curves across outer iterations and a controlled comparison replacing Shortcut Trajectory with PLTM under identical matched modules.
  • Adding schedule controls, regularization experiments on the original-trajectory baseline, and measured per-iteration costs with estimated distillation runtimes.

The revisions are highlighted in blue for easy reference.

We would appreciate your feedback on whether these clarifications and additional results address your concerns. If any questions remain, we would be happy to clarify them during the discussion period.

Thank you for your time and consideration. We look forward to hearing your thoughts.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.