- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper identifies and formalizes a failure mode of dataset distillation that is invisible to standard per-distillate evaluation: two synthetic sets can each reproduce training on their own source exactly (zero endpoint error), yet their mass-weighted union fails to reproduce training on the union of the real sources. The authors call this the composition gap.
Core contributions:
-
Composability–affinity principle (Prop. 3.3): exact composition of equal-mass pairs for all pairs is equivalent to the compression map being affine (midpoint-Jensen).
-
One-dimensional exact separation (Thm. C.1): the finite-step clock g_p(a) = (1-(1-\eta a)^p)/\eta is strictly concave in curvature, so compress-then-mix undershoots mix-then-compress even when both constituents are exact.
-
Exact matrix law (Thm. 4.2): for two-to-one compression, \Phi_ind(\theta_0) - \Phi_R(\theta_0) = \eta^2(V_H\theta_0 - w), with V_H the mass-weighted Hessian variance, giving \mathcal{E}_ind = \eta^4[tr(V_H^2\Sigma_0) + ||V_H\mu_0 - w||_2^2] and the common-optimum specialization \mathcal{E}_ind = \eta^4 tr(V_H^2 M_0).
-
Arbitrary ratio p = T/K (Thm. E.2) with a noncommutative remainder bound and a lower bound via strong monotonicity of matrix powers.
-
Detectability certificate (Cor. 4.3) comparing a structure-only estimate \mathcal{C}_struct against the weighted constituent error \mathcal{U}_src, plus an HVP-computable estimator P_comp.
-
Experiments: learned quadratic distillates recover the closed form to machine precision once n_syn >= d; on a ConvNet the local prediction matches \Delta\Phi at small step size (cosine 1.000, calibration within ~3%); and in the practical DM regime the source residual, not the structural term, dominates, with the two terms anti-aligned.
The paper is unusually careful: assumptions are stated explicitly, negative results are reported (Cor. 4.3's inequality is satisfied for 0/96 initializations in the endpoint-matching setting), the flow-limit boundary is proved (Prop. E.3), and the no-control-from-individual-errors result (Prop. F.3) closes the obvious loophole.
Critical strengths
S1. "Each distillate is faithful, the union is not" is the failure mode federated distillation and synthetic-data marketplaces actually hit. Fidelity as a property of the union rather than of its constituents is the contribution here, and the negative result is worth more than a positive one would have been.
S2. Theorem 4.2 needs neither commutativity nor a common optimum for Eq. 5:
V_H = \sum_i \alpha_i (H_i - \bar H)^2 = (1/2)\sum_{i,j}\alpha_i\alpha_j (H_i - H_j)^2 \succeq 0,
with the affine term w = \sum_i \alpha_i H_i b_i - \bar H \bar b separated, and a zero-error condition (V_H\Sigma_0^{1/2}=0, V_H\mu_0=w) sharp enough to read off. That agreement on the optimum does not imply agreement at the endpoint is the insight; Fig. 2(c) shows it with identical spectra and rotated eigenbases.
S3. The \eta^4 scaling is tested (3.71–3.96 for smooth activations, ReLU's 3.49 put down to non-smoothness), the closed form holds on learned quadratic sets (Table 2, ratio 1.000 \pm 0.000 at n_syn >= d), and the breakdown at \eta L \approx 1 is reported rather than hidden.
S4. Eq. 14, d_ind(\theta_0) = \Delta\Phi(\theta_0) + r_src(\theta_0), is exact to 6\times10^{-8} and does the diagnostic job: which term dominates where, rather than the structural term explaining practical failures.
S5. Scoping is honest: the limitations section, Fig. 7's zero-crossing, the corollary called a diagnostic rather than a certificate on networks, and the short-schedule, feature-skew and five-source results labelled preliminary.
S6. Table 3 and Prop. H.2 (Ky Fan) separate what scalar retiming can fix from what it cannot:
\mathcal{E}(\tau) \geq \sum_{j=r+1}^{d} s_j(B)^2, \quad B = (I - e^{-TH_R})M_0^{1/2}.
Critical weaknesses
W1. The title names a mechanism that the paper's own experiments say is not operative in the regime it measures. At the DM operating point C_struct sits 13–24× below U_src (Table 6), Cor. 4.3's condition fails at 0/96 initializations, and eta^2 r_comp is nearly orthogonal to d_ind (cosine 0.01–0.06, Fig. 3b); the excess is carried by source residuals. Lead the abstract with the two-regime story, and move the +6.6 no-heterogeneity control out of J.7. It has nothing to do with V_H and it is the better result.
W2. The accuracy gain cannot be attributed as presented. Table 7 gains 5.3–5.9 points at every heterogeneity level, IID included; warm start recovers 0.44 of a 6.15-point gap; the allocation control falls from 4.5 to 1.0 and from 5.4 to 3.4. That points to coverage rather than composition. The joint set also receives the summed budget, IPC 15–20 against 10 per source, and the caption should say so.
W3. Quadratics to networks is asserted more than validated. delta_H is 0.14–0.29 over two steps, which is not small, and the 3.75–3.88 exponent shortfall is put down to it without an experiment that varies delta_H independently. A baseline predicting Delta-Phi from tr(V_H) alone would isolate whether the eigenstructure matters on a network and not only in the quadratic construction.
Questions for authors
-
On the accuracy gain: Given that the gain persists with no split (+6.6), is reduced but not eliminated by class-allocation matching (+1.0/+3.4 vs +4.5/+5.4 on two partitions), and is only 0.44/6.15 recovered by warm-starting from the union — what is the mechanism? Is the claim that joint distillation produces a better optimization landscape for the evaluation network rather than a better endpoint map? If so, the endpoint and accuracy results are measuring different things, and the paper should say so explicitly in Sec. 7. A concrete experiment: evaluate the joint vs. union sets on the same endpoint target but with different evaluation seeds to see if the accuracy gap tracks endpoint gap at all (the paper has the data — Table 7 shows it does not at IID).
-
On the matrix-vs-scalar test on networks: Can the authors report the correlation of \Delta\Phi with a scalar-curvature-only predictor (e.g., \eta^2 tr(V_H)(\theta_0-\theta^*) projected appropriately) alongside P_comp? Table 4 gives Spearman \rho = 0.991 for P_comp and 0.955 for P_iso (curvature-only), with partial \rho dropping to 0.316 for P_iso — this is suggestive that the affine/eigenvector structure matters, but a direct scalar baseline is missing.
-
On \delta_H: The Hessian changes 14–29% over two steps. Does the \eta^4 exponent improve if the local model is re-centered (frozen quadratic at \theta_1 rather than \theta_0)? This would test whether the shortfall is the frozen-Hessian approximation or something else, and would strengthen the Sec. 5 bridge.
-
On the arbitrary-p bound: Since Fig. 7 shows the certified region is \eta L \lesssim 0.12 at moderate rank, and the practical regime is \eta L \approx 1, is there a version of the lower bound that uses the exact p=2 identity iteratively (two-step windows) to certify positivity at larger \eta L? If not, the paper should state plainly that the lower bound is a small-step result and the large-step evidence is empirical.
-
On budget matching for m=5: Can the five-source experiment be rerun with total budget fixed (e.g., IPC 10 total split across m sources, or IPC 10 per source with joint budget also scaled)? The current confound makes the "gap grows with m" claim weak.
6 comments
Response to Reviewer 3r7X (2/2): W3, Q2, Q3, W4, Q4, W6
W3, Q2 (matrix versus scalar prediction). Added (Appendix L.1, summarized in Section 6). Predictors of the measured discrepancy ΔΦ at on the SiLU and tanh networks:
| Predictor | Cosine with ΔΦ | Norm relative to ‖ΔΦ‖ |
|---|---|---|
| Matrix (Theorem 4.2) | 0.999–1.000 | 1.00–1.03 |
| Scalar curvature, | −0.03 to 0.08 | below 2 × 10⁻⁴ |
| Per-layer curvature | 0.01 to 0.05 | below 4 × 10⁻⁴ |
For two sources any scalar predictor is a multiple of , so we added the per-layer predictor, which is not. The matrix structure carries the prediction.
W3, Q3 ( and the exponent shortfall). The shortfall does not come from the expansion point: re-centring the quadratic model at or at the midpoint leaves the prediction's exponent at 3.98–4.00 in every cell. It is the drift of the calibration ratio with ηL, and it closes as the step size decreases: the local exponent at SiLU Dir-0.1 rises from 3.05 to 3.91 across the window and reaches 3.95 and 3.98 at two further halvings in double precision. The revision replaces the attribution to a marginally satisfied A4 by this statement. In this design depends on alone, so it could not be varied independently.
W4 (theoretical statements). Proposition 3.3: the sentence after it attributes it to the midpoint-Jensen equation (Aczél, 1966); the contribution is the application. Theorem E.2: Section 4.3 now states that for p > 2 the bound certifies positivity only at small ηL and that the experiments rely on the exact p = 2 identity. Lemma D.2(ii) is qualified to the oracle gap. Corollary 4.3 and : the observable-gap bound (Corollary H.3) does not use the common-optimum hypothesis; on a network its remainder is unavailable, so is used there as a diagnostic and not as a certificate, which Section 5 now says.
Q4 (iterating the p = 2 identity). Two exact statements replace the lower bound, which is a small-step result. Proposition E.4 gives the K-step difference of the composed and joint maps as an identity, for any ratio and without a common optimum. Its limit is the new Corollary E.7: training on a union of exact distillates converges to the stationary point , which differs from the union optimum by to first order in and tends to the weighted average of the source optima as p → ∞. The error therefore does not accumulate without bound. On a linear probe with exact distillates it costs 6.5 ± 3.1 and 8.9 ± 2.8 accuracy points at Dir-0.1 for p = 512 and in the limit, and nothing at IID (Figure 5c, Table 18).
W6 (presentation). Figure 2(a) shows one level set per Hessian with the endpoints and the displacement, Table 6 is split into two blocks, the notation table covers the experimental symbols, and a new Figure 5 shows the matched comparison of DM and MTT and the linear-probe result. The supplement includes the code of the submitted and of the additional experiments, with per-partition and row-level results under results/rebuttal/.
Thanks for the response and the budget-matched experiment, it answers my main concern. The proportional-allocation row is the decisive one: +11.5 points for classes held by one party against -2.2 for shared classes tells me the joint-set gain was mostly budget, which is what I suspected. The W1 reframing and the negative result on Q1 (p = 0.15, 0.53) are both fine by me. I'm still lukewarm on the "beyond drift" contribution in W5, but 78% of the excess removed at 0.005 source-error change is concrete enough that I won't hold the paper on it. Score: {{2}} → {{3}}. Sorry for the extra experiment.
On process, since the PCs asked: I used an LLM to help draft this review and to check the numbers I quoted against your tables. The reading, the criticisms and the score are mine.
Response to Reviewer 3r7X (1/2): W2, Q5, W1, W5, Q1
We thank the reviewer for the careful reading. References are to the revised PDF.
W2, Q5 (budget confound; budget-matched sources). We ran the budget-matched experiment, and the reviewer's reading is correct for label-skewed DM sources. With 200 images allocated over sources and classes in proportion to the real counts, the gain of the joint set at Dir-0.1 falls from 5.3 ± 0.7 to 0.7 ± 1.6 points for two sources (Table 21). The effect is visible per class:
A class held by one party has ten images in the original union against 15 to 19 in the budget-matched joint set. The endpoint excess of DM falls with the allocation as well. Section 6 now states that the DM gain is not claimed as a composition phenomenon, and the caption of Figure 4 states that the joint set receives the sum of the two budgets.
The experiment also shows what the budget does not explain. At the same allocation, budget (150 images) and partitions, trajectory matching retains a gap of 16.6 ± 3.8 points at Dir-0.1, against 1.3 ± 0.4 for DM and 2.8 ± 0.3 for trajectory matching at IID (Table 24, Figure 5a). For five DM sources 3.7 ± 2.0 points remain under the proportional allocation, positive on every partition. Without heterogeneity, where the allocation is already matched, DM keeps a gain of 5.3 to 6.6 points, about half of which conditioning the second set on the first recovers (Table 20).
W1 (framing). Adopted. The abstract, the third contribution, Section 6 and Section 7 now state that for the learned sets we tested the source residual is the larger term, and they name the sources of the accuracy gain. Section 6 also states what the residual offsets: under label skew the local discrepancy of the real sources is 66% to 108% of the total endpoint error of the joint set, against 0.2% at IID (Table 7).
W5 (client drift; composition-aware objective). For two local steps the FedAvg drift and ΔΦ are the same vector (Section 4.2), the Cloc column of Table 7 gives its value on our partitions, and the stationary point of Corollary E.7 is the known FedAvg fixed point. Beyond drift the paper contributes the exact matrix form with its affine term, the decomposition dind=ΔΦ+rsrc, and the static-data setting. A composition-aware objective removes 78% of the excess over the joint set on the network at a relative source-error change of 0.005 (Table 19).
Q1 (does accuracy track the excess). Across the 20 CIFAR-10 partitions we find no significant association (ρ = 0.15, p = 0.53), and across the two SVHN levels the two rise together (ρ = 0.83 over 10 partitions). Under intervention they separate: conditioning raises the accuracy of the union at every level without lowering its excess (Table 20). Section 7 therefore states training fidelity and downstream accuracy as distinct requirements.