acceptodds
Under review as a conference paper at ICLR 2027

Flow Matching for Soft-Label Debiasing in Long-Tailed Dataset Distillation

Abstract

Dataset distillation compresses large-scale datasets into compact synthetic data to reduce storage and training costs. In long-tailed scenarios, soft labels generated during distillation suffer from entangled bias originating from the imbalanced distillation model and synthetic images. Existing calibration methods typically assume this bias as a linear additive component and apply static statistical shifts to the logits. However, the distribution shift induced by class imbalance is highly nonlinear, rendering linear adjustments insufficient for disentangling semantic relationships from frequency-induced bias. This paper proposes Flow Matching Label Transport (FMLT), a framework that models soft-label debiasing as a continuous transport process on probability manifolds. We formulate the transition from biased soft-label distributions to unbiased target distributions using ordinary differential equations. By learning a time-dependent vector field conditioned on synthetic image features, FMLT transports the biased labels along nonlinear trajectories without relying on additive assumptions. Experiments on CIFAR-10/100-LT and ImageNet-1k-LT demonstrate that FMLT consistently improves performance across various distillation baselines and data budget settings. In extreme long-tailed scenarios, the nonlinear transport mechanism restores inter-class semantic topologies and improves tail-class accuracy.

Primary area:
generative models
Keywords:
Dataset Distillation, Long-Tailed Distribution, Flow Matching, Soft-Label Debiasing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2oQpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies soft-label bias in long-tailed dataset distillation. The authors argue that existing calibration methods, which shift logits by a class-dependent constant, are too rigid to undo the kind of distortion that class imbalance introduces into soft labels. Instead, they propose FMLT, which treats debiasing as a transport problem: a small MLP, conditioned on the synthetic image's features, learns a velocity field in logit space via conditional flow matching, and the biased logits are pushed toward a target distribution by integrating this field with an Euler solver. The method is a post-hoc module that plugs into existing distillation pipelines such as SRe2L, G-VBSM and EDC, and is evaluated on CIFAR-10/100-LT and ImageNet-1k-LT against the ADSA calibration baseline.

Critical strengths

Soft-label bias in long-tailed dataset distillation is a real and fairly under-explored problem, and I appreciate that the method is designed as a lightweight post-hoc step that does not require touching the distillation pipeline itself. It is cheap to run and the authors do try it on top of several different distillation methods, including at ImageNet scale. The general intuition that a per-sample, feature-dependent correction might do better than a single global shift is reasonable and could be worth exploring properly.

Critical weaknesses

The method does not actually do what the paper claims. The whole paper is built around the argument that a linear, additive logit shift is not enough and that a nonlinear, feature-conditioned transport is needed. But the target distribution in Eq. 2 is just the original prediction reweighted by the inverse of each class's mean confidence. In logit space, this is exactly a class-wise constant shift, i.e., the same kind of linear correction the paper criticizes, with the class prior replaced by the mean confidence. Since the reference path is a straight line between the original and target logits, the velocity the network is trained to regress is essentially the same constant vector for every sample, regardless of the image features or the time step. So there is no meaningful nonlinear trajectory for the MLP to learn, and the feature conditioning has nothing to adapt to. Following this through, the step-count ablation does not make sense either: for a constant velocity field, a single Euler step is already exact, so accuracy should not keep improving with more steps. The text also disagrees with itself about where the performance plateaus, and the default number of steps is not the best one in the table.

Key baselines and ablations are missing. The target in Eq. 2 can be computed in closed form for every synthetic image, and the flow is trained and applied on the same set of images. The obvious question is what happens if you simply use Eq. 2 as the soft label directly, without any flow. I suspect this would perform the same, which would mean that any gain over ADSA comes from using mean confidence instead of the class prior, not from flow matching. There is also no ablation removing the image-feature conditioning, even though this is the central selling point, and the repeated claims about "restoring inter-class semantic topology" are not backed by any measurement at all.

The experimental evidence is thin and has consistency issues. No standard deviations or numbers of runs are reported, and many of the improvements over ADSA are around one point or less, which is hard to interpret without variance estimates. The gains are also strikingly uniform across very different settings, especially in the ImageNet-LT table. In that table, some rows cannot be reconciled with each other: for SRe2L at one IPC setting, the head accuracy drops sharply after calibration while the overall accuracy goes up, and I could not find any split of head/mid/tail classes under which both the baseline and the calibrated rows would be consistent. In addition, the text makes claims that the tables do not support, e.g., that EDC with FMLT outperforms the other baselines in all settings. There is only a single ablation in the whole paper, and the main text does not even fill the page limit, so there was clearly room for the experiments that are missing.

The theory adds length but not insight, and contains errors. Most of the appendix restates textbook results (Picard–Lindelöf, Euler truncation error, universal approximation, Rademacher bounds) that say nothing specific about this method. Some of it is also wrong or inconsistent. Lemma 5 claims the target yields a uniform class marginal, directly contradicting the main text, which says it does not. The Fisher–Rao distance is misidentified as the square root of the JS divergence. Theorem 1 claims cosine similarity is invariant under a shift, and its proof then admits it is not. Theorem 3 assumes the MLP output is bounded, which does not hold for a ReLU/GELU network taking unbounded logits as input. Theorem 4 bounds a Euclidean kinetic energy using a Fisher–Rao argument without connecting the two.

Presentation is poor. Figure 1 is a conceptual illustration with no real data behind it, and Figure 2 is a low-resolution raster plot where the axis labels and legends are basically unreadable; its legend also says "Ours" instead of FMLT, and it uses fractions while the tables use percentages. The reference list contains several duplicated entries for the same papers, and citations are formatted inconsistently throughout (e.g., "Jiang et al. Jiang et al. (2025)"). Overall, the paper reads as an unfinished draft rather than a submission ready for ICLR.

Questions for authors
  1. Since the target in Eq. 2 amounts to a class-wise constant shift in logit space, the regression target for the vector field should be (up to a softmax-invariant offset) the same constant for every sample. Can you show that the learned field is actually non-constant and genuinely depends on the image features? For example, how much does the predicted velocity vary across samples and time steps?

  2. What happens if you use the closed-form target from Eq. 2 directly as the soft label, without training or integrating any flow? Please report this baseline in the main tables. If it performs about the same as FMLT, what does the flow-matching component contribute?

  3. Please add an ablation that removes the image-feature conditioning h(x) from the vector field network. This is the central claim of the paper, but it is never tested.

  4. With a straight-line reference path and a constant target velocity, one Euler step should already be exact. Why does accuracy keep improving as the number of steps grows in Table 3? Also, the main text and Appendix D.4 disagree on where performance plateaus, and the default setting is not the best one in the table. Which is correct?

  5. In Table 2, the overall accuracy for SRe2L and SRe2L + ADSA at IPC=50 does not seem consistent with the reported head/mid/tail accuracies under any fixed class split. How exactly are the head, mid and tail groups defined, and how is overall accuracy computed? Could you double-check these numbers?

  6. How many runs are the reported results averaged over, and what are the standard deviations? Many of the gains over ADSA are around one point or less, so variance estimates are needed to judge whether they are meaningful.

  7. Were the ADSA results produced by your own runs with the tuning protocol described in Appendix D.3, or taken from the original paper? If the latter, were the experimental settings identical?

  8. Lemma 5 states that the target produces a uniform class marginal, while Section 3.2 says it does not in general. Which statement is correct, and how should the proof be read?

  9. The paper repeatedly claims that FMLT "restores inter-class semantic topology." Can you provide a quantitative measure that supports this, for example some comparison of inter-class similarity structure before and after calibration?

Reviewer_t73WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the Flow Matching Label Transport (FMLT), a post-hoc soft-label calibration method for long-tailed dataset distillation. The method addresse the bias caused by imblanced traning dats, and existing methods typically correct using additive logits adjustments. FMLT formulates label caliberation as as continuous transport process. First it constructs target soft-label distributions using inverse class-wise confidence reweighting and after that learns an image-conditioned vector field through conditional flow matching to transport the original logits toward the calibrated targets. Then the resulting soft labels are used to train downstream classification models.

The authors evaluate the method on CIFAR-10-LT, CIFAR-100-LT and ImageNet-1k-LT across multiple dataset distillation baselines and synthetic data budgets. Results show a consistent improvement over ADSA, specifically in tail class accuracy under severe class imbalance.

Critical strengths
  1. The method FMLT consistenly outperforms ADSA across CIFAR-10-LT, CIFAR-100-LT, and ImageNet-1k-LT under different imbalance factors and data budgets.

  2. The method improves tail class performance and it can be easily integrated into exiisting dataset distillation methods without modifying their training pipelines.

Critical weaknesses
  1. Equation (2) already defines the calibrated target soft labels, which can be used directly. The paper does not show comparison of this direct caliberation with FMLT and hence it makes it difficult to determine are the shown improvements come from flow matching or simply from the target label construction.

  2. The ablation studies do not sufficiently isolate the contribution of the individual components of the method.

  3. Most improvements over ADSA are approximately 1 percentage point, but the presented tables do not include standard deviations, without this, it is difficult to assess whether these gains are reliable.

Questions for authors
  1. Since Equation (2) already provides the calibrated target soft labels, could the authors explain why flow matching is needed? A comparison with directly using these target labels would help clarify the benefit of the proposed approach.

  2. Could the authors provide more detailed ablation studies to clarify the contributions of individual components, particularly image-feature conditioning and the learned flow-matching mechanism?

  3. Could the authors compare FMLT with a simpler nonlinear calibration network without ODE integration? This would help determine whether continuous transport provides advantages over conventional nonlinear calibration.

  4. Most improvements over ADSA are around 1 percentage point. Could the authors report standard deviations across multiple independent seeds to demonstrate the reliability of these improvements?

Reviewer_LaPQOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a dataset distillation method called Flow Matching Label Transport (FMLT). It stems from the commonly utilized assumption of additive and stationary logit shifts, arguing which is insufficient to represent the non-linear deformation between datasets before and after the distillation. The proposed FMLT then consider the debiasing progress as a continuous transport, constructing the soft target via reweighting the teacher predictions with a finite sample sets, and subsequently using flow matching to train the mapping. Experimental results involve its integration on multiple baselines such as SRe2L, GVBSM, and EDC, and are evaluated across CIFAR-10-LT, CIFAR100-LT, and ImageNet-1k-LT. Consistent improvements can be achieved in comparison with previous SOTA ADSA.

Critical strengths

The research topic on soft-label bias in dataset distillation is relevant, and the mentioned limitation of existing linear-based additive assumptions exists. The proposed FMLT is a flexible add-on that can be easily combined with other methods with trivial overhead, and the empirical improvements, as mentioned, are consistent across baselines and benchmarks.

Critical weaknesses
  • Target defined in Eq. 2 is in fact an additive logit shift also, which contradicts with the core claim. Starting from the incorrect definition of the confidence score in line 215, the score should be written as $ \bar{p}(y=i) $ after the expectation over xx is taken across all samples and no longer depends on the image being calibrated. In this case, the likelihood ratio between arbitrary classes is simply recalibrated by a universal ratio (e.g., for any j,kj,k, ptarget(y=j∣x)ptarget(y=k∣x)=pˉ(y=k)pˉ(y=j)\frac{p_{target}(y=j|x)}{p_{target}(y=k|x)} = \frac{\bar{p}(y=k)} {\bar{p}(y=j)}), leading to the same additive logit correction as z1−z0=−log⁡pˉz_1-z_0=-\log\bar{p}. This hurts the core of this manuscript, as it can be represented by a constant class-wise correction without image-dependent non-linear modeling as claimed in the intro section.
  • Unclear role of flow matching. Given the unbiasing as an image-independent additive correction (z1−z0=−log⁡pˉz_1-z_0=-\log\bar{p}), we can directly have a closed-form solution for the linear path as ut=−log⁡pˉu_t=-\log\bar{p} in Eq. 5. In this case, it does not seem necessary to implement a neural network to map the path, and the conditioning on feature h(x)h(x) has no contribution.
  • Limited contributions. Even without the first two major concerns, the current manuscript seems to apply flow matching in dataset distillation directly, without much task-specific modification.
Questions for authors
  • Could the authors clarify whether the pˉ\bar{p} in Eq. 2 and line 215 is a class-wise constant? If not, please introduce the correct implementation. Otherwise, could you elaborate on how you implement non-linear dibiasing with logit rescaling using this class-wise constant ratio?
  • What happens if FMLT discards the h(x) conditioning, or uses the closed-form solution (z0−log⁡pˉz_0 - \log\bar{p}) directly without network modeling?
  • In line 215, it is said that the core pˉ\bar{p} is estimated using synthetic data as a held-out set. Would such an estimation be sensitive to cases with a limited synthetic data budget?
  • This is more like a minor suggestion. The current citation style does not seem aligned with the official one used in ICLR (probably the manuscript uses \cite rather than \citep in this submission version).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.