acceptodds
Under review as a conference paper at ICLR 2027

The Composition Gap in Dataset Distillation

Abstract

Dataset distillation compresses a training set into a small synthetic set, usually evaluated one at a time. In federated and data-governance settings, several parties distill their own data and a user trains on their union. We ask whether the union of separately distilled sets reproduces training on the union of the real data (composability) and show that it can fail even when every source is distilled exactly and the total budget admits an exact joint distillate. Compressing a training trajectory into fewer steps transforms the source statistics nonlinearly, so averaging compressed sources differs from compressing their average. For quadratic objectives we derive the exact composition error for two-to-one step compression in terms of the source-Hessian variance and the linear terms of the losses, and on a smooth network at small step sizes this prediction captures the local endpoint discrepancy in magnitude and direction. For learned synthetic sets, however, the composed error decomposes exactly into this local discrepancy and an aggregate source residual. Under endpoint matching the residual exceeds the structural term by more than an order of magnitude, and under distribution matching the two terms partly cancel. Joint distillation also retains an accuracy advantage when both sets are distilled from the same dataset, where the local discrepancy is exactly zero. Training fidelity and downstream accuracy are therefore distinct requirements, neither established by evaluating each set on its own. Code will be released.

Primary area:
learning theory
Keywords:
Dataset distillation, composability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3r7XOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies and formalizes a failure mode of dataset distillation that is invisible to standard per-distillate evaluation: two synthetic sets can each reproduce training on their own source exactly (zero endpoint error), yet their mass-weighted union fails to reproduce training on the union of the real sources. The authors call this the composition gap.

Core contributions:

  1. Composability–affinity principle (Prop. 3.3): exact composition of equal-mass pairs for all pairs is equivalent to the compression map being affine (midpoint-Jensen).

  2. One-dimensional exact separation (Thm. C.1): the finite-step clock g_p(a) = (1-(1-\eta a)^p)/\eta is strictly concave in curvature, so compress-then-mix undershoots mix-then-compress even when both constituents are exact.

  3. Exact matrix law (Thm. 4.2): for two-to-one compression, \Phi_ind(\theta_0) - \Phi_R(\theta_0) = \eta^2(V_H\theta_0 - w), with V_H the mass-weighted Hessian variance, giving \mathcal{E}_ind = \eta^4[tr(V_H^2\Sigma_0) + ||V_H\mu_0 - w||_2^2] and the common-optimum specialization \mathcal{E}_ind = \eta^4 tr(V_H^2 M_0).

  4. Arbitrary ratio p = T/K (Thm. E.2) with a noncommutative remainder bound and a lower bound via strong monotonicity of matrix powers.

  5. Detectability certificate (Cor. 4.3) comparing a structure-only estimate \mathcal{C}_struct against the weighted constituent error \mathcal{U}_src, plus an HVP-computable estimator P_comp.

  6. Experiments: learned quadratic distillates recover the closed form to machine precision once n_syn >= d; on a ConvNet the local prediction matches \Delta\Phi at small step size (cosine 1.000, calibration within ~3%); and in the practical DM regime the source residual, not the structural term, dominates, with the two terms anti-aligned.

The paper is unusually careful: assumptions are stated explicitly, negative results are reported (Cor. 4.3's inequality is satisfied for 0/96 initializations in the endpoint-matching setting), the flow-limit boundary is proved (Prop. E.3), and the no-control-from-individual-errors result (Prop. F.3) closes the obvious loophole.

Critical strengths

S1. "Each distillate is faithful, the union is not" is the failure mode federated distillation and synthetic-data marketplaces actually hit. Fidelity as a property of the union rather than of its constituents is the contribution here, and the negative result is worth more than a positive one would have been.

S2. Theorem 4.2 needs neither commutativity nor a common optimum for Eq. 5:

V_H = \sum_i \alpha_i (H_i - \bar H)^2 = (1/2)\sum_{i,j}\alpha_i\alpha_j (H_i - H_j)^2 \succeq 0,

with the affine term w = \sum_i \alpha_i H_i b_i - \bar H \bar b separated, and a zero-error condition (V_H\Sigma_0^{1/2}=0, V_H\mu_0=w) sharp enough to read off. That agreement on the optimum does not imply agreement at the endpoint is the insight; Fig. 2(c) shows it with identical spectra and rotated eigenbases.

S3. The \eta^4 scaling is tested (3.71–3.96 for smooth activations, ReLU's 3.49 put down to non-smoothness), the closed form holds on learned quadratic sets (Table 2, ratio 1.000 \pm 0.000 at n_syn >= d), and the breakdown at \eta L \approx 1 is reported rather than hidden.

S4. Eq. 14, d_ind(\theta_0) = \Delta\Phi(\theta_0) + r_src(\theta_0), is exact to 6\times10^{-8} and does the diagnostic job: which term dominates where, rather than the structural term explaining practical failures.

S5. Scoping is honest: the limitations section, Fig. 7's zero-crossing, the corollary called a diagnostic rather than a certificate on networks, and the short-schedule, feature-skew and five-source results labelled preliminary.

S6. Table 3 and Prop. H.2 (Ky Fan) separate what scalar retiming can fix from what it cannot:

\mathcal{E}(\tau) \geq \sum_{j=r+1}^{d} s_j(B)^2, \quad B = (I - e^{-TH_R})M_0^{1/2}.

Critical weaknesses

W1. The title names a mechanism that the paper's own experiments say is not operative in the regime it measures. At the DM operating point C_struct sits 13–24× below U_src (Table 6), Cor. 4.3's condition fails at 0/96 initializations, and eta^2 r_comp is nearly orthogonal to d_ind (cosine 0.01–0.06, Fig. 3b); the excess is carried by source residuals. Lead the abstract with the two-regime story, and move the +6.6 no-heterogeneity control out of J.7. It has nothing to do with V_H and it is the better result.

W2. The accuracy gain cannot be attributed as presented. Table 7 gains 5.3–5.9 points at every heterogeneity level, IID included; warm start recovers 0.44 of a 6.15-point gap; the allocation control falls from 4.5 to 1.0 and from 5.4 to 3.4. That points to coverage rather than composition. The joint set also receives the summed budget, IPC 15–20 against 10 per source, and the caption should say so.

W3. Quadratics to networks is asserted more than validated. delta_H is 0.14–0.29 over two steps, which is not small, and the 3.75–3.88 exponent shortfall is put down to it without an experiment that varies delta_H independently. A baseline predicting Delta-Phi from tr(V_H) alone would isolate whether the eigenstructure matters on a network and not only in the quadratic construction.

Questions for authors
  1. On the accuracy gain: Given that the gain persists with no split (+6.6), is reduced but not eliminated by class-allocation matching (+1.0/+3.4 vs +4.5/+5.4 on two partitions), and is only 0.44/6.15 recovered by warm-starting from the union — what is the mechanism? Is the claim that joint distillation produces a better optimization landscape for the evaluation network rather than a better endpoint map? If so, the endpoint and accuracy results are measuring different things, and the paper should say so explicitly in Sec. 7. A concrete experiment: evaluate the joint vs. union sets on the same endpoint target but with different evaluation seeds to see if the accuracy gap tracks endpoint gap at all (the paper has the data — Table 7 shows it does not at IID).

  2. On the matrix-vs-scalar test on networks: Can the authors report the correlation of \Delta\Phi with a scalar-curvature-only predictor (e.g., \eta^2 tr(V_H)(\theta_0-\theta^*) projected appropriately) alongside P_comp? Table 4 gives Spearman \rho = 0.991 for P_comp and 0.955 for P_iso (curvature-only), with partial \rho dropping to 0.316 for P_iso — this is suggestive that the affine/eigenvector structure matters, but a direct scalar baseline is missing.

  3. On \delta_H: The Hessian changes 14–29% over two steps. Does the \eta^4 exponent improve if the local model is re-centered (frozen quadratic at \theta_1 rather than \theta_0)? This would test whether the shortfall is the frozen-Hessian approximation or something else, and would strengthen the Sec. 5 bridge.

  4. On the arbitrary-p bound: Since Fig. 7 shows the certified region is \eta L \lesssim 0.12 at moderate rank, and the practical regime is \eta L \approx 1, is there a version of the lower bound that uses the exact p=2 identity iteratively (two-step windows) to certify positivity at larger \eta L? If not, the paper should state plainly that the lower bound is a small-step result and the large-step evidence is empirical.

  5. On budget matching for m=5: Can the five-source experiment be rerun with total budget fixed (e.g., IPC 10 total split across m sources, or IPC 10 per source with joint budget also scaled)? The current confound makes the "gap grows with m" claim weak.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 3r7X (1/2): W2, Q5, W1, W5, Q1

We thank the reviewer for the careful reading. References are to the revised PDF.

W2, Q5 (budget confound; budget-matched sources). We ran the budget-matched experiment, and the reviewer's reading is correct for label-skewed DM sources. With 200 images allocated over sources and classes in proportion to the real counts, the gain of the joint set at Dir-0.1 falls from 5.3 ± 0.7 to 0.7 ± 1.6 points for two sources (Table 21). The effect is visible per class:

Joint minus union, accuracy pointsClasses held by one partyShared classes
Ten images per represented class+11.5 ± 2.1+2.7 ± 3.0
Proportional allocation−2.2 ± 0.7+1.5 ± 1.7

A class held by one party has ten images in the original union against 15 to 19 in the budget-matched joint set. The endpoint excess of DM falls with the allocation as well. Section 6 now states that the DM gain is not claimed as a composition phenomenon, and the caption of Figure 4 states that the joint set receives the sum of the two budgets.

The experiment also shows what the budget does not explain. At the same allocation, budget (150 images) and partitions, trajectory matching retains a gap of 16.6 ± 3.8 points at Dir-0.1, against 1.3 ± 0.4 for DM and 2.8 ± 0.3 for trajectory matching at IID (Table 24, Figure 5a). For five DM sources 3.7 ± 2.0 points remain under the proportional allocation, positive on every partition. Without heterogeneity, where the allocation is already matched, DM keeps a gain of 5.3 to 6.6 points, about half of which conditioning the second set on the first recovers (Table 20).

W1 (framing). Adopted. The abstract, the third contribution, Section 6 and Section 7 now state that for the learned sets we tested the source residual is the larger term, and they name the sources of the accuracy gain. Section 6 also states what the residual offsets: under label skew the local discrepancy of the real sources is 66% to 108% of the total endpoint error of the joint set, against 0.2% at IID (Table 7).

W5 (client drift; composition-aware objective). For two local steps the FedAvg drift and ΔΦ are the same vector (Section 4.2), the Cloc\mathcal C_{\rm loc} column of Table 7 gives its value on our partitions, and the stationary point of Corollary E.7 is the known FedAvg fixed point. Beyond drift the paper contributes the exact matrix form with its affine term, the decomposition dind=ΔΦ+rsrcd_{\rm ind}=\Delta\Phi+r_{\rm src}, and the static-data setting. A composition-aware objective removes 78% of the excess over the joint set on the network at a relative source-error change of 0.005 (Table 19).

Q1 (does accuracy track the excess). Across the 20 CIFAR-10 partitions we find no significant association (ρ = 0.15, p = 0.53), and across the two SVHN levels the two rise together (ρ = 0.83 over 10 partitions). Under intervention they separate: conditioning raises the accuracy of the union at every level without lowering its excess (Table 20). Section 7 therefore states training fidelity and downstream accuracy as distinct requirements.

AuthorsOpenReview ↗

Response to Reviewer 3r7X (2/2): W3, Q2, Q3, W4, Q4, W6

W3, Q2 (matrix versus scalar prediction). Added (Appendix L.1, summarized in Section 6). Predictors of the measured discrepancy ΔΦ at ηref\eta_{\rm ref} on the SiLU and tanh networks:

PredictorCosine with ΔΦNorm relative to ‖ΔΦ‖
Matrix (Theorem 4.2)0.999–1.0001.00–1.03
Scalar curvature, tr(Hi)/d\mathrm{tr}(H_i)/d−0.03 to 0.08below 2 × 10⁻⁴
Per-layer curvature0.01 to 0.05below 4 × 10⁻⁴

For two sources any scalar predictor is a multiple of g1−g2g_1-g_2, so we added the per-layer predictor, which is not. The matrix structure carries the prediction.

W3, Q3 (δH\delta_H and the exponent shortfall). The shortfall does not come from the expansion point: re-centring the quadratic model at θ1\theta_1 or at the midpoint leaves the prediction's exponent at 3.98–4.00 in every cell. It is the drift of the calibration ratio with ηL, and it closes as the step size decreases: the local exponent at SiLU Dir-0.1 rises from 3.05 to 3.91 across the window and reaches 3.95 and 3.98 at two further halvings in double precision. The revision replaces the attribution to a marginally satisfied A4 by this statement. In this design δH\delta_H depends on η\eta alone, so it could not be varied independently.

W4 (theoretical statements). Proposition 3.3: the sentence after it attributes it to the midpoint-Jensen equation (Aczél, 1966); the contribution is the application. Theorem E.2: Section 4.3 now states that for p > 2 the bound certifies positivity only at small ηL and that the experiments rely on the exact p = 2 identity. Lemma D.2(ii) is qualified to the oracle gap. Corollary 4.3 and PcompP_{\rm comp}: the observable-gap bound (Corollary H.3) does not use the common-optimum hypothesis; on a network its remainder is unavailable, so PcompP_{\rm comp} is used there as a diagnostic and not as a certificate, which Section 5 now says.

Q4 (iterating the p = 2 identity). Two exact statements replace the lower bound, which is a small-step result. Proposition E.4 gives the K-step difference of the composed and joint maps as an identity, for any ratio and without a common optimum. Its limit is the new Corollary E.7: training on a union of exact distillates converges to the stationary point (∑iαifp(Hi))−1∑iαifp(Hi)θi⋆\big(\sum_i\alpha_i f_p(H_i)\big)^{-1}\sum_i\alpha_i f_p(H_i)\theta_i^\star, which differs from the union optimum by (p−1)η2Hˉ−1(VHθ⋆−w)\tfrac{(p-1)\eta}{2}\bar H^{-1}(V_H\theta^\star-w) to first order in η\eta and tends to the weighted average of the source optima as p → ∞. The error therefore does not accumulate without bound. On a linear probe with exact distillates it costs 6.5 ± 3.1 and 8.9 ± 2.8 accuracy points at Dir-0.1 for p = 512 and in the limit, and nothing at IID (Figure 5c, Table 18).

W6 (presentation). Figure 2(a) shows one level set per Hessian with the endpoints and the displacement, Table 6 is split into two blocks, the notation table covers the experimental symbols, and a new Figure 5 shows the matched comparison of DM and MTT and the linear-probe result. The supplement includes the code of the submitted and of the additional experiments, with per-partition and row-level results under results/rebuttal/.

Thanks for the response and the budget-matched experiment, it answers my main concern. The proportional-allocation row is the decisive one: +11.5 points for classes held by one party against -2.2 for shared classes tells me the joint-set gain was mostly budget, which is what I suspected. The W1 reframing and the negative result on Q1 (p = 0.15, 0.53) are both fine by me. I'm still lukewarm on the "beyond drift" contribution in W5, but 78% of the excess removed at 0.005 source-error change is concrete enough that I won't hold the paper on it. Score: {{2}} → {{3}}. Sorry for the extra experiment.

On process, since the PCs asked: I used an LLM to help draft this review and to check the numbers I quoted against your tables. The reading, the criticisms and the score are mine.

Reviewer_wNUmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper's discussion starts from a practical problem of dataset distillation in the scenario of federate learning. When distilled datasets are from data holders, the model trained on the union of distilled dataset underperforms that trained on jointly distilled datasets. The paper investigate the reason of composition discrepancy.

Critical strengths
  1. The paper details the reason of the gap between the union of distilled set and the training on union of the real data. The proof is detailed and convincing.
  2. Experiment design is thoughtful, which covers different datasets, distillation methods, model architectures, the number of sources, etc. The conclusion remain consistent with large amount of experiment results.
Critical weaknesses
  1. The paper only raise the reason of failure, but doesn't provide a method to address it. I'd like to hear from authors what directions this paper could provide to address the problem.
  2. The paper only test Distribution Matching (DM) under official schedule, while DM is not gradient or trajectory matching based. Experiments with DC or MTT could better support the claim.
  3. I suggest authors to add a table of the meaning of frequently used symbols, to help audiences understand the work. Also, try to avoid using too many symbols in the experiment section.
Questions for authors

In future work, the authors plan to extend the experiment to ImageNet scale. May I ask what could that bring to the current conclusion besides the scalability?

3 comments
AuthorsOpenReview ↗

Response to Reviewer wNUm

We thank the reviewer for the careful reading. References are to the revised PDF.

W1 (no method; which directions). The revision adds three directions, each with a measurement:

DirectionWhat it needsMeasured effect
Composition-aware objective (Appendix L.3, Table 19)joint optimization of the two sets, no pooled dataremoves 60%, 78% and 85% of the excess endpoint error on the network at λ = 1, 3 and 10; relative source error 0.385 → 0.390 at λ = 3
Sequentially conditioned distillation (Appendix L.4, Table 20)the second party's own data and the first party's released setrecovers 43% of the accuracy gap with no split, 57% on IID partitions and 62% after a second round
Count-proportional budget allocation (Appendix L.5, Table 21)the class counts of the sourcesDM gap at Dir-0.1 falls from 5.3 ± 0.7 to 0.7 ± 1.6 points for two sources and 1.2 ± 1.2 for three

For parties that cannot coordinate, the decomposition gives a validation rule: measure the union against a reference on the union, since the fidelity of each constituent certifies neither the endpoint nor the accuracy of the combination.

W2 (only DM). We added trajectory matching (MTT) and gradient matching (DC). MTT is compared with DM at the same allocation, budget (150 images) and partitions, three per level (Table 24, Figure 5a,b):

Joint minus union, accuracy pointsIIDDir-0.1
MTT2.8 ± 0.316.6 ± 3.8
DM6.9 ± 1.41.3 ± 0.4

The MTT excess endpoint error is positive at Dir-0.1 under every calibration reading, and its mean is negative at IID. DC at its official schedule (44.6% on the full training set against the published 44.9%) replicates the DM endpoint findings: the excess is larger at Dir-0.1 than at IID on every partition (1.88 ± 0.91 against 0.49 ± 0.31 × 10⁻³, Table 23).

W3 (symbols). The notation table (Appendix B, Table 1) now covers every measured quantity of Section 6, the new tables name their quantities in words, and a new Figure 5 shows the matched comparison of DM and MTT.

Q (what would ImageNet add). Two tests and no new mechanism: whether the residual-dominated regime persists when the classes are many and the per-class budget is small, and whether the allocation share of the gap changes when skew spreads over hundreds of classes. The linear-probe experiment already uses ImageNet-pretrained features, and there the structural term costs up to 8.9 points at Dir-0.1 and nothing at IID (Figure 5c).

Thank authors for the detailed rebuttal and additional experiments. While a few limitations remain, I appreciate these improvements and will maintain my rating at 3 (Weak Acceptance).

We sincerely thank the reviewer for reading our response and the additional experiments, and for the positive assessment. Your questions on remedies and on other distillation methods led to the composition-aware objective and the trajectory-matching and gradient-matching experiments, which strengthened the paper. We agree that limitations remain. They are stated in Section 7 and Appendix K, and we will keep them explicit in the final version. We will discuss and address these limitations in future work. Thank you again for your time and for the constructive review.

Reviewer_b5k2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper asks whether the union of separately distilled datasets can replace training on the union of the real sources. The main theoretical point is that compressing T training steps into fewer synthetic steps transforms the source curvature nonlinearly, so averaging compressed sources is not the same as compressing their average. For quadratic objectives and two-to-one compression this gives an exact error formula in terms of the variance of the source Hessians plus an affine term, even when each source is distilled exactly. On a smooth network at small step sizes the quadratic prediction matches the local discrepancy well. For learned synthetic sets, however, the observed error is dominated by the per-source distillation error rather than the structural term, and the accuracy advantage of joint distillation persists even when both sets come from the same data, where the structural term is zero. The authors conclude that endpoint fidelity and downstream accuracy are separate requirements.

Critical strengths

The question is relevant for federated and multi-party uses of dataset distillation, where people do combine independently distilled sets, and the paper gives a clean and exact answer in the quadratic case. The derivation is careful and does not need commuting Hessians.

I also appreciate how honest the empirical part is. The authors test their own mechanism against controls (the no-split control, the class-allocation-matched joint set, the exact decomposition into local discrepancy and source residual) and report clearly when it does not explain what they observe.

Critical weaknesses
  1. The mechanism the paper develops does not seem to matter in practice, by the paper's own results. On networks, the source distillation error is much larger than the structural term under endpoint matching, and under DM the two partly cancel. The accuracy gain of joint distillation is about the same whether the structural term is large or exactly zero (the no-split control). So the theory explains an effect that is mostly invisible in the settings people care about, and the practically relevant gap seems to come from something else, e.g. redundancy between two sets optimized without knowledge of each other, or class allocation. That second finding is interesting, but it is not really explained here and is close to what one would expect: a single DM run with 20 images per class should beat two independent runs with 10 each.

  2. The theoretical contribution is modest relative to the amount of formalism. For one-step synthetic training, the defect is essentially the client-drift term from local updates in federated optimization, which the paper itself acknowledges. The rest is a Jensen-type gap for a concave matrix function, plus bounds that become loose at the step sizes actually used. The endpoint metric is also a very short-horizon proxy: two real steps vs. one synthetic step. DM sets are not trained for this, so they need a per-set step rescaling to be evaluated at all, and the excess error changes a lot depending on whether this calibration is applied. I am not convinced that T=2 endpoint fidelity tells us much about composability over a full training run.

The paper is also hard to read. There are many closely related quantities (C_loc, C_struct, P_comp, U_src, two versions of the excess), and the main text leans heavily on a very long appendix. No remedy is proposed: the composition-aware objective is left to future work, and post-hoc reweighting does not change accuracy.

Questions for authors
  1. Does the structural term become important at longer horizons or larger compression ratios, closer to how distilled sets are actually used? Any evidence beyond (T=4, K=2) would help.

  2. Have you tried a trajectory-matching method (e.g. MTT), whose objective is much closer to the endpoint maps the theory is about? That seems the most natural place for the mechanism to show up.

  3. What explains the accuracy gain in the no-split control? Is it more than the expected advantage of one jointly optimized set over two redundant ones? For example, does the gap shrink if the two independent runs are made more diverse (different real initializations, or a diversity term)?

  4. Could you try even a small-scale version of the composition-aware objective? It would show whether the theory leads to anything actionable.

  5. Why is the per-set scalar step calibration the right protocol for DM sets, and which reading (calibrated or not) should we trust for the excess endpoint error?

  6. Beyond the transformed statistics, what does the analysis add over the client-drift analysis of Li et al. (2020) for K=1?

6 comments
AuthorsOpenReview ↗

Response to Reviewer b5k2 (1/3): W1, W2, Q6, W3

We thank the reviewer for the careful reading. References are to the revised PDF.

W1 (the mechanism does not drive the practical results). The two-to-one law is exact for quadratic objectives and quantitatively accurate on smooth networks at small step sizes (cosine 0.999–1.000, norm ratio 1.00–1.03; Appendix L.1). The new results show a persistent accuracy cost in an exact-distillation probe and a substantial local discrepancy at the practical DM operating point.

(i) Corollary E.7 shows that exact quadratic distillates can encode a biased stationary point in the released data. When this bias is nonzero, training longer on their union does not recover the real-union optimum. Exact source fidelity alone does not guarantee composability.

(ii) With exact distillates of real features, a linear probe loses 6.5 ± 3.1 accuracy points at p = 512 and 8.9 ± 2.8 in the limit under Dir-0.1 skew, and nothing at IID (Figure 5c; table in Q1). On the network the real discrepancy grows 15- to 76-fold between 2 and 16 steps.

(iii) At the two-step operating point of DM, the measured local discrepancy of the real sources is comparable to the total endpoint error of the jointly distilled set under label skew and negligible at IID (Table 7, now stated in Section 6):

Endpoint error (× 10⁻³)IIDDir-1.0Dir-0.5Dir-0.1
Local discrepancy of the real sources0.0164.855.588.09
Jointly distilled set7.347.347.337.50
Ratio0.2%66%76%108%

With three and five sources the ratio is 1.6 and 1.9 (Table 21). Source sets that reproduced their own maps exactly would leave the union with an error of this size, whereas an exact joint set has none. The source residual and the measured local discrepancy are comparable in scale and partly cancel; their vector interaction matters for the composed endpoint error.

On accuracy we agree with the reviewer's specific point: for the learned DM sets the gain of joint distillation comes mostly from two other effects, which the revision identifies and quantifies. Under label skew a count-proportional allocation reduces the gain from 5.3 ± 0.7 to 0.7 ± 1.6 points (Table 21), and without heterogeneity conditioning the second set on the first recovers about half of it (Q3). The theory also supplies a corrective target: in the quadratic control admitting exact source distillation, the plain surrogate leaves the normalized composition error at 1.00, while the corrected surrogate reduces it to 0.18 at λ = 10 (Q4).

W2, Q6 (novelty relative to client drift). For two local steps, FedAvg client drift (Li et al., 2020) and our ΔΦ are the same vector, which Section 4.2 now states, and the stationary point of Corollary E.7 is the known FedAvg fixed point (Pathak & Wainwright, 2020; Charles & Konečný, 2021), which we cite. The paper adds: (i) the static-data setting, in which two exact distillates realize the displacement without a federated training protocol; (ii) the exact matrix form with its affine term, which the network prediction needs (scalar and per-layer curvature predictors have norms below 4 × 10⁻⁴ of ‖ΔΦ‖); (iii) the decomposition dind=ΔΦ+rsrcd_{\rm ind}=\Delta\Phi+r_{\rm src} for learned sets; (iv) a composition-aware objective that needs no pooled data (Q4). On the bounds for other ratios, Section 4.3 now states that they certify positivity only at small ηL and that the experiments rely on the exact two-to-one identity. The two-step horizon and the step calibration are addressed in Q1 and Q5.

W3 (readability, no remedy). The notation table covers every measured quantity of Section 6, Table 6 is split into two blocks, Figure 2(a) is simplified, and a new Figure 5 shows the matched comparison of DM and MTT. For remedies see Q3 and Q4.

AuthorsOpenReview ↗

Response to Reviewer b5k2 (2/3): Q1, Q2

Q1 (longer horizons, larger ratios). Corollary E.7 gives the exact answer in the quadratic setting. Training on a union of exact distillates converges to θind=(∑iαifp(Hi))−1∑iαifp(Hi)θi⋆\theta_{\rm ind}=\big(\sum_i\alpha_i f_p(H_i)\big)^{-1}\sum_i\alpha_i f_p(H_i)\theta_i^\star, displaced from the union optimum by (p−1)η2Hˉ−1(VHθ⋆−w)\tfrac{(p-1)\eta}{2}\bar H^{-1}(V_H\theta^\star-w) to first order in η\eta, and at fixed η\eta it tends to the weighted average of the source optima as p → ∞. The displacement vanishes under a common optimum, and any nonzero displacement persists under continued training.

We measured its cost on a least-squares probe on frozen ImageNet ResNet-18 features of CIFAR-10, where exact distillates exist in closed form (Figure 5c, Table 18). Accuracy lost by the union relative to the exact joint distillate, in points, five partitions per level:

Levelp = 128p = 512p → ∞
IID0.0 ± 0.00.0 ± 0.10.0 ± 0.1
Dir-1.00.4 ± 0.30.8 ± 0.61.0 ± 0.8
Dir-0.51.5 ± 0.82.8 ± 1.53.5 ± 1.9
Dir-0.13.2 ± 2.96.5 ± 3.18.9 ± 2.8

The loss is negligible at IID at every ratio and grows with skew and with p. On the network the real T-step discrepancy grows 15- to 76-fold between T = 2 and T = 16 at Dir-0.1 (Appendix L.1). The leading term of Theorem 4.2 is a small-ηT statement and overestimates at longer horizons, which Section 4.3 now says.

Q2 (trajectory matching). We ran MTT and compared it with DM at the same allocation, budget (150 images) and partitions, three per level (Table 24, Figure 5a,b):

Gain at IIDGain at Dir-0.1Excess at IID (× 10⁻³)Excess at Dir-0.1 (× 10⁻³)
MTT2.8 ± 0.316.6 ± 3.8−0.8 ± 1.3+3.1 ± 1.7
DM6.9 ± 1.41.3 ± 0.4+0.1 ± 0.10.0 ± 0.2

Gain is the accuracy of the joint set minus that of the union, in points; the excess endpoint error uses one step multiplier for the union. The MTT excess is positive on every Dir-0.1 partition under all three calibration readings and above that of every IID partition. At matched allocation and budget, MTT retains a much larger gap on the three label-skewed partitions than on the three IID partitions. Allocation alone therefore does not remove the MTT gap. MTT uses the proportional allocation throughout, since its official recipe does not train with ten images per class on highly skewed sources. Gradient matching replicates the DM endpoint pattern (Table 23) and keeps 2.2 ± 1.2 points under the proportional allocation.

AuthorsOpenReview ↗

Response to Reviewer b5k2 (3/3): Q3, Q4, Q5

Q3 (what explains the no-split gain). Missing coordination, about half of which a second party recovers from the first party's released set. The second set is distilled with the first frozen and the DM loss evaluated on the union (Table 20; accuracy in percent):

SettingIndependent unionConditioned unionJoint setGap recovered
No split46.4 ± 0.549.1 ± 0.352.6 ± 0.443% (62% after a second round)
IID46.7 ± 0.849.6 ± 0.551.8 ± 0.857%

The conditioned union lies above the independent union on every replicate. Despite starting from different real images, the independent runs overlap: the per-class effective rank of the union is 8.9 against 10.3 for the joint set, and 9.5 after conditioning (Table 16). Under label skew conditioning acts only on the classes that both parties hold, and the allocation is the effective remedy (W1).

Q4 (a small composition-aware objective). Jointly optimizing the two sets with λ times the squared endpoint error of the union added to the two source objectives removes most of the excess over the joint set on the network at a small cost in source fidelity (Appendix L.3, Table 19; Dir-0.1):

λ01310
Excess over the joint set (× 10⁻³)0.2890.1170.0630.042
Excess removed–60%78%85%
Relative source error0.3850.3860.3900.398

No pooled data is needed: the union's target can be replaced by the average of the parties' own real maps, with or without the correction η²r_comp, which each party computes from its own data, the exchanged mean gradient and one Hessian-vector product. On the network, both the plain and the corrected surrogate targets give this improvement. In the quadratic control admitting exact source distillation, the plain target leaves the normalized composition error at 1.00, whereas the theory-corrected target reduces it to 0.18 at λ = 10.

Q5 (the per-set calibration). The conclusion does not rest on the per-set calibration. We report three readings (Table 22): the per-set multiplier of the submission, one multiplier per level fitted on held-out initializations and frozen, and none. Excess of the DM unions on CIFAR-10 under the original allocation (× 10⁻³):

ReadingIIDDir-1.0Dir-0.5Dir-0.1
Per-set0.071.421.422.38
Frozen0.020.510.650.97
None0.000.080.170.28

The excess is positive on every skewed partition under the two calibrated readings and on fourteen of fifteen without calibration, and indistinguishable from zero at IID under all three. The calibration changes the magnitude, and a two-factor check shows that the difference between the calibrated readings comes mostly from the scaling unit, one multiplier per source set or one for the union (Appendix L.6).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the revision

Dear AC and Reviewers,

We sincerely thank the AC and all reviewers for the time and care devoted to our submission and for the constructive reviews. Changes in the revised PDF are in blue. Four results answer the central questions.

1. The structural term matters in practice (b5k2 W1, Q1; 3r7X W1). A union of sets that reproduce the p-step maps of their sources exactly takes, at every step, the average of those maps. We evaluated this update directly on the ConvNet, with no distillation and therefore no source residual (Appendix L.2):

Accuracy lost against training on the real union, pointsp = 8p = 32p = 128p = 512
IID−0.8 ± 0.3−0.9 ± 0.3−0.9 ± 0.3−0.7 ± 0.3
Dir-0.12.3 ± 0.25.2 ± 0.87.4 ± 1.69.9 ± 2.5

There is no average loss at IID. On every Dir-0.1 partition the loss is positive and increases with p, and four times the training length does not remove it (6.4 ± 1.3 points at p = 512). A linear probe with exact distillates loses 6.5 ± 3.1 points at the same ratio (Corollary E.7, Figure 5c). The quadratic law is quantitatively supported on smooth networks at small step sizes (cosine 0.999–1.000, norm ratio 1.00–1.03), and at the DM operating point the measured local discrepancy under label skew is 66% to 108% of the total endpoint error of the jointly distilled set, against 0.2% at IID (Table 7).

2. At a matched budget the gap depends on the method (b5k2 Q2, wNUm W2, 3r7X W2). Trajectory matching (MTT) and distribution matching (DM) at the same count-proportional allocation, budget (150 images) and partitions (Figure 5a, Table 24):

Joint minus union, accuracy pointsIIDDir-0.1
MTT2.8 ± 0.316.6 ± 3.8
DM6.9 ± 1.41.3 ± 0.4

For DM the allocation removes most of the gap under label skew (5.3 ± 0.7 to 0.7 ± 1.6 points, Table 21), which answers the budget question of 3r7X.

3. Remedies exist (wNUm W1, b5k2 Q3, Q4). A composition-aware objective removes 78% of the excess endpoint error on the network at a relative source-error change of 0.005 and needs no pooled data (Table 19). Conditioning the second set on the first party's released set recovers 43% to 62% of the accuracy gap without heterogeneity (Table 20).

4. Robustness (b5k2 Q5, 3r7X W3). Under the original DM allocation, the mean excess under label skew remains positive under all three calibration readings (Table 22). Gradient matching reproduces the endpoint pattern under both calibrated readings (Table 23), and scalar and per-layer curvature baselines have norms below 4 × 10⁻⁴ of the measured discrepancy in the small-step network tests (Appendix L.1).

Scope. The theorems and the experiments of the submission are kept. New: Corollary E.7, Figure 5 and Appendix L (Tables 18–24). The abstract, the third contribution and Section 6 now separate the sources of the accuracy gain. The appendix figures shift by one number. The scripts and per-partition results of the additional experiments are in the supplement.

Dear reviewers,

This is a kind reminder that the discussion with the authors ends on 18th Nov. Please take a moment to leave a comment to acknowledge the authors’ responses and engage in the discussion if you have further questions, particularly on points that may affect your assessment.

Thank you!

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.