acceptodds
Under review as a conference paper at ICLR 2027

Spectral Gradient Surgery for Domain Generalizable Dataset Distillation

Abstract

Dataset Distillation (DD) synthesizes a compact synthetic dataset that preserves the training utility of a full dataset. However, its standard formulation assumes that test data follow the same distribution as training data, which rarely holds in practice. A straightforward extension—applying post-hoc Domain Generalization (DG) techniques to distilled data—is ill-suited because existing DG methods rely on the natural diversity of real datasets, which compact synthetic sets inherently lack, and their augmentation overhead conflicts with the efficiency goal of DD. We therefore study Domain Generalizable Dataset Distillation (DGDD), a setting in which distilled datasets are evaluated on target domains unseen during distillation, through the lens of Distribution Matching (DM). We hypothesize that the OOD vulnerability of DM stems from compressing domain-consistent and domain-specific information indiscriminately, and propose Spectral Gradient Surgery (SGS). SGS computes domain-wise DM gradients with respect to the synthetic images, transforms them into the Fourier domain, and measures per-frequency phase agreement across source domains. It then augments the standard DM update with two terms: a class-relevant component that emphasizes frequency components consistent across source domains, and a domain-relevant component that preserves source-domain variation to diversify the synthetic set. Across three benchmarks of increasing resolution, SGS consistently improves the average OOD accuracy of two DM methods without modifying their objectives or downstream training cost, at the price of additional offline distillation cost.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Dataset Distillation, Domain Generalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2GCoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The submission proposes SGS for DD across domains. It computes DM gradients for each source, then uses Fourier phase agreement to form a consensus update, and adds a domain-specific residual for each synthetic image. Experiments on Digits-DG, PACS, and CORe50-Hard show improved OOD accuracy with DM and HDD-DM. The paper also studies pseudo-domains, component ablations, gradient alignment, and distillation cost.

Critical strengths

SGS consistently improves DM and HDD-DM across the evaluated benchmarks and budgets without changing downstream classifier training. The pseudo-domain results further support its applicability when domain labels are unavailable. Controls with matched gradient norms, individual components, and pixel-space agreement suggest that the gains are not simply due to increasing the update size.. The cost analysis also helps assess the additional work required during distillation.

Critical weaknesses

I find the results promising, but the explanation of why SGS works needs to be more precise:

  • Phase agreement is a useful heuristic, but it does not necessarily identify transferable class information. Shared nuisance signals can pass the filter, while spatial shifts can reduce agreement for the same pattern. A large gradient from one domain can also dominate the score. The random-phase analysis does not establish when these effects matter, so the claimed class/domain separation needs stronger support.
  • The domain residual in eq9 is gs−mean(gs)g_s − mean(g_s). Its reported zero similarity with target gradients needs clarification, since averaging the residuals first would cancel them.
  • The efficiency evidence is also limited: the main setting has higher distillation cost, while the comparison at approximately matched compute covers only a single-source setting. I encourage the authors to clarify the diagnostic and separate downstream efficiency from total distillation cost.
Questions for authors
  1. What allows SGS to distinguish transferable class information from a class-correlated nuisance shared by all source domains?
  2. Fourier phase changes under translation. How does this affect the consensus update when the same useful pattern appears at different positions across domains?
  3. Could you compare against an equal-weighted phase score to separate cross-domain agreement from dominance by one large gradient?
  4. Is the residual–target similarity in tab7 computed per domain or after averaging residuals? The residuals sum to zero by construction, so the aggregation order is essential to interpreting this diagnostic.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 2GCo (1)

We thank the reviewer for the meaningful and valuable comments.

W1 / Q1. Shared class-correlated nuisance

We acknowledge that SGS cannot isolate semantic class information but instead identifies information shared across the source domains. Thus, we have replaced "class-relevant" with "source-consistent" throughout the manuscript. However, note that this limitation is common to other invariance-based methods, such as IRM, Fish, and Fishr. Nevertheless, the source-consistent component is empirically better aligned with unseen domains: gclassg^{\text{class}} is closer to the held-out target gradient than gg on every PACS target (cosine 0.27 vs. 0.21 on average; Table 9).

We have discussed this limitation in Appendix A.

W1 / Q2. Translation and the consensus update

We agree that translating an image changes its Fourier phases. However, SGS applies the Fourier transform not to real images but to domain-wise gradients, all computed with respect to the same synthetic image and network. Thus, image-level translation has a limited effect on phase agreement because real images influence these gradients only through the class-mean embedding of each domain, which aggregates them into a coarse feature map. We verified this through controlled shifts, independently translating the real images of each source domain in a random direction at the beginning of distillation and reporting the ranges across target domains.

Δr\Delta r when real images are shifted by2 px4 px8 px16 px
Digits-DG (32 px)−0.01 to +0.05−0.01 to +0.12−0.05 to +0.17−0.15 to +0.26
PACS (64 px)−0.01 to +0.02−0.02 to +0.05−0.04 to +0.09−0.10 to +0.09
CORe50-H (128 px)0.00 to +0.00+0.00 to +0.01+0.01 to +0.02+0.03 to +0.06

The mean rr remains stable in most settings; the largest change occurs for the 16 px shift on Digits-DG, which corresponds to half of the image width. Furthermore, although Digits-DG does not allow a controlled pose study, CORe50 objects are dynamically moved during capture, so their position and pose vary across images and sessions. SGS still improves DM on CORe50-Hard from 27.4 to 28.9 (Table 1).

We added these results in Appendix C.3 (Table 14).

W1 / Q3. Dominating domain, Comparison with equal-weighted scoring

We first measured how often one domain dominates the score using real DM gradients at IPC=10 at the beginning of distillation, reporting the ranges across the four target domains of each dataset.

Digits-DGPACSCORe50-Hard
Domain grad norm ratio1.3–1.71.3–1.71.2–1.3
Actual consensus91–97%95–98%92–94%

Domain grad norm ratio denotes the median over classes of the max/min source-domain gradient norm, and Actual consensus denotes, among the frequency components with r>0.8r>0.8, the percentage in which every domain is within 90° of the mean phase. These results suggest that:

  • Dominance by one domain is rare: The gradient norms of the source domains are similar, with a median max/min ratio below 2.
  • High-agreement components reflect broadly shared directions: Over 90% of the components with r>0.8r>0.8 exhibit agreement across all domains.

We added these results in Section 4.4 (Table 8).

We replaced rr with an equal-vote score reqr^{eq}, computed in the same way as rr but from unit-length phase vectors so that every domain receives the same weight, and kept everything else unchanged (Digits-DG, IPC=10).

Digits-DGMNVYAvg
DM85.351.453.669.965.1
SGS with rr (Eq. 8)90.354.155.371.067.7
SGS with reqr^{eq}89.752.154.870.966.9

These results show that the gain of SGS does not depend on magnitude dominance: SGS with reqr^{eq} still improves DM on all four targets (65.1 to 66.9 on average). Nevertheless, the magnitude can be helpful for weighting the update: with unit-length phase vectors, a low-magnitude coefficient that contributes little to the update receives the same vote as a dominant one, so components that should be suppressed can be emphasized, which may explain the remaining gap to rr.

We discussed these results in Section 4.3 ("Equal-vote Agreement Score") and Table 5.

AuthorsOpenReview ↗

Response to Reviewer 2GCo (2)

W1. Scope of the random-phase analysis

We agree that Appendix C establishes attenuation only under the random-phase condition and does not establish when the three effects raised in W1 matter. A nuisance shared by all sources is preserved in the same way as class information, which we now state as a limitation (Q1). For the other two cases, the diagnostics on real DM gradients in Q2 and Q3 show that translations barely change rr except for extreme shifts and that one domain rarely dominates.

We discussed this in Section 4.4 and Appendix C.3.

W2 / Q4. Aggregation in Table 7

We acknowledge the reviewer's concern. The reported value was the signed cosine averaged over the three sources and all synthetic images of all classes. Since the residuals sum to zero, this average largely cancels. Therefore, the near-zero entries did not indicate orthogonality. As reported below, when evaluated per source, the residuals are not orthogonal to the held-out target gradient.

Target (sources 1/2/3)Source 1Source 2Source 3
Digits M (N/V/Y)+0.64−0.62−0.25
Digits N (M/V/Y)+0.18−0.20−0.11
Digits V (M/N/Y)−0.61+0.03+0.67
Digits Y (M/N/V)−0.49−0.04+0.51
PACS P (A/C/S)+0.57+0.08−0.47
PACS A (P/C/S)+0.54+0.12−0.51
PACS C (P/A/S)−0.02+0.12−0.04
PACS S (P/A/C)−0.27−0.12+0.27
CORe50-H s5 (s7/s9/s10)−0.02−0.03+0.03
CORe50-H s7 (s5/s9/s10)−0.20+0.15+0.05
CORe50-H s9 (s5/s7/s10)−0.10+0.28−0.16
CORe50-H s10 (s5/s7/s9)−0.02+0.18−0.14

Each entry denotes the cosine similarity between the residual gdomaing^{\text{domain}} of one source and the gradient of the held-out target. For example, with M as the target, the residual of N has a similarity of +0.64 and that of V has −0.62. We observe that the signs differ across sources on every target. The residual of a source close to the target points toward it, while that of a distant source points away. This aligns with the visual similarity between domains: M and N in Digits-DG, and P and A in PACS, are highly similar. This also suggests the role of gdomaing^{\text{domain}}: assigning the source domains uniformly across the synthetic samples diversifies the synthetic set, which may help cover unseen domains.

We have revised the discussion and Table 7 (now Table 9) in Section 4.4 ("Alignment with Unseen Domains"), and added the full table above as Table 17 (Appendix D.1).

AuthorsOpenReview ↗

Response to Reviewer 2GCo (3)

W3. Downstream efficiency and distillation cost

We consistently distinguish between the two throughout the manuscript:

  • Downstream training cost: It is the same as DM, because IPC, resolution, and the training procedure do not change.
  • Offline distillation cost: It scales roughly with the number of source domains, about 3.25× DM with three sources (Table 6).

We also extended the compute-matched comparison to the main multi-source setting by running DM for 66k iterations, which matches the wall-clock cost of SGS.

IPC=10CostPACSCORe50-Hard
DM (20k iter.)1.0×35.627.4
DM (66k iter.)3.3×35.027.5
SGS (20k iter.)3.25×38.628.9

Running DM 3.3× longer barely changes accuracy on either benchmark, so the gain of SGS is not explained by additional distillation compute. This comparison is added to Appendix D (Table 16).

We are glad to provide further detail on any of these points.

Reviewer_yFyWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies dataset distillation for unseen domains. It creates a small synthetic training set from known source domains. A model trained on this set is then tested on a domain not used during distillation. The proposed method, Spectral Gradient Surgery (SGS), computes gradients for the synthetic images separately for each source domain. It applies a Fourier transform to these gradients. It then adds two terms to the standard distribution-matching update. One term captures signals shared across domains. The other keeps differences between domains. SGS improves the average unseen-domain accuracy of DM and HDD-DM on various benchmarks. It adds no cost to downstream training, but makes distillation more expensive.

Critical strengths

SGS is easy to add to distribution-matching methods. The gains are fairly consistent across different models, datasets, and synthetic-set sizes. The experiments include results from five runs, component ablations, a pixel-space comparison, gradient alignment with unseen domains, and distillation cost. These tests help explain how the method works.

Critical weaknesses
  1. The closest prior work is not discussed or compared. [1] already study the same setting: creating synthetic data from known domains and testing on an unseen domain. Their method, CADM, uses the same PACS leave-one-domain-out setting. It also tests more datasets and transfer across model architectures. Their published results cannot be compared directly with Table 1 because the backbone and training setup differ. Still, without this comparison, it is unclear what is new about the DGDD setting and how SGS compares with the closest method. The paper should discuss this work and clarify its new contribution. It should also compare SGS with CADM using the same source splits, images per class, data augmentation, backbone, and optimization budget.

  2. High phase agreement does not always mean that a signal is class-related. There are two basic problems. First, moving an image pattern changes its Fourier phases. The same class pattern at different positions can therefore have low phase agreement. This raises a concern about Section 3.3, which presents spectral comparison as a way to handle spatial misalignment. Second, Eq. 8 gives more weight to larger coefficients. It does not give each domain an equal vote. For example, consider three coefficients: 100∠0,1∠π,1∠π.100\angle 0,\quad 1\angle\pi,\quad 1\angle\pi. They give (r=98/102\approx0.96), even though two of the three domains have the opposite phase. Table 11 compares spectral agreement with a pixel-sign method, but it does not test these two cases. The paper should test controlled shifts, pose changes, and differences in gradient scale. It should also compare with methods that use unit-length phase vectors, align phases, or measure spatial correlation. These tests are needed to support the interpretation of the shared signal.

  3. Transfer to different model architectures is not tested. The synthetic sets are tested only with shallow ConvNets from the same family used in the pipeline. These networks have three, four, or five layers, depending on the dataset. The results do not show whether the synthetic data work well for substantially different models. The gains may depend on the architecture used during distillation. This is a common concern in dataset distillation, and CADM [1] directly tests it. The paper should create each synthetic set once with the stated backbone, then train different models, such as ResNet and VGG, from scratch on that same set. Ideally, it should also test a larger or more complex benchmark. These results are needed to support a broad claim about domain-generalizable synthetic data.

Questions for authors

Please see the Weakness part.

5 comments
AuthorsOpenReview ↗

Response to Reviewer yFyW (1)

We thank the reviewer for the meaningful and valuable comments.

W1. Relation to CADM [1]

We clarify the novel contributions of SGS compared to CADM as follows:

  • Gradient-level analysis of DM under domain shift: CADM identifies domain-sensitive feature channels through pretrained domain discriminators. We instead analyze the update that DM applies to each synthetic image, measure agreement across source domains directly from their gradients with a per-frequency phase-agreement score, and decompose the update accordingly into source-consistent and domain-specific components.

  • Preserving domain-specific information: While CADM suppresses domain-sensitive channels during distillation, we find that domain-specific signals may also help diversify the synthetic images. Per-source analysis (Table 17) shows that the domain-specific component of a source close to the held-out domain points toward that domain's gradient, so it may carry information relevant to unseen domains.

  • Pretraining-free and plug-in compatible: SGS needs no pretrained expert or domain discriminators, whereas CADM trains them for each leave-one-domain-out split. SGS uses the same randomly initialized networks as DM and changes only the update rule, so it applies directly to existing DM-based methods.

  • Applicability without domain labels: CADM relies on domain labels to train its domain discriminators. In contrast, SGS can operate without domain labels by using the proposed K-means pseudo-domains. Accordingly, SGS extends to single-source DG, while CADM is evaluated only in the multi-source setting.

Since the official implementation of CADM is not publicly available, we attempted to reproduce it based on the paper. However, the paper alone does not provide sufficient implementation details for reliable reproduction such as (1) distillation: the number of iterations, optimizer, learning rate, and batch sizes; (2) evaluation: the image resolution, optimizer, learning rate, and number of epochs; and (3) proposed components: implementation details, including the pretraining setup.

In the revised manuscript, we have revised Sections 1, 2, and 3.1 to clarify our novel contributions relative to CADM.

[1] Liu et al., Channel-masked Asymmetric Distribution Matching for Cross-Domain Generalized Dataset Distillation, AAAI 2026.

AuthorsOpenReview ↗

Response to Reviewer yFyW (2)

W2. High phase agreement and class information

We acknowledge that SGS cannot isolate semantic class information but instead identifies information shared across the source domains. Thus, we have replaced "class-relevant" with "source-consistent" throughout the manuscript.

Effect of image-level translation. We agree that translating an image changes its Fourier phases. However, SGS applies the Fourier transform not to real images but to domain-wise gradients, all computed with respect to the same synthetic image and network. Thus, image-level translation has a limited effect on phase agreement because real images influence these gradients only through the class-mean embedding of each domain, which aggregates them into a coarse feature map. We verified this through controlled shifts, independently translating the real images of each source domain in a random direction at the beginning of distillation and reporting the ranges across target domains.

Δr\Delta r when real images are shifted by2 px4 px8 px16 px
Digits-DG (32 px)−0.01 to +0.05−0.01 to +0.12−0.05 to +0.17−0.15 to +0.26
PACS (64 px)−0.01 to +0.02−0.02 to +0.05−0.04 to +0.09−0.10 to +0.09
CORe50-H (128 px)0.00 to +0.00+0.00 to +0.01+0.01 to +0.02+0.03 to +0.06

The mean rr remains stable in most settings; the largest change occurs for the 16 px shift on Digits-DG, which corresponds to half of the image width. Furthermore, although Digits-DG does not allow a controlled pose study, CORe50 objects are dynamically moved during capture, so their position and pose vary across images and sessions. SGS still improves DM on CORe50-Hard from 27.4 to 28.9 (Table 1).

We added these results in Appendix C.3 (Table 14).

Magnitude dominance and gradient scale. Although the example (100, −1, −1) is extreme, the analysis is correct: Eq. 8 can reward a single dominant domain. To assess the practical significance of this limitation, we evaluate the sensitivity of SGS to such dominance using real DM gradients at IPC=10 at the beginning of distillation, reporting the ranges across the four target domains of each dataset.

Digits-DGPACSCORe50-Hard
Domain grad norm ratio1.3–1.71.3–1.71.2–1.3
Actual consensus91–97%95–98%92–94%

Domain grad norm ratio denotes the median over classes of the max/min source-domain gradient norm, and Actual consensus denotes, among the frequency components with r>0.8r>0.8, the percentage in which every domain is within 90° of the mean phase. These results suggest that:

  • Dominance by one domain is rare: The gradient norms of the source domains are similar, with a median max/min ratio below 2.
  • High-agreement components reflect broadly shared directions: Over 90% of the components with r>0.8r>0.8 exhibit agreement across all domains.

We added these results in Section 4.4 (Table 8).

Unit-length phase. We replaced rr with an equal-vote score reqr^{eq}, computed in the same way as rr but from unit-length phase vectors so that every domain receives the same weight, and kept everything else unchanged (Digits-DG, IPC=10).

Digits-DGMNVYAvg
DM85.351.453.669.965.1
SGS with rr (Eq. 8)90.354.155.371.067.7
SGS with reqr^{eq}89.752.154.870.966.9

These results show that the gain of SGS does not depend on magnitude dominance: SGS with reqr^{eq} still improves DM on all four targets (65.1 to 66.9 on average). Nevertheless, the magnitude can be helpful for weighting the update: with unit-length phase vectors, a low-magnitude coefficient that contributes little to the update receives the same vote as a dominant one, so components that should be suppressed can be emphasized, which may explain the remaining gap to rr.

We discussed these results in Section 4.3 ("Equal-vote Agreement Score") and Table 5.

Phase alignment and spatial correlation. For each pair of domain gradients of the same synthetic image on Digits-DG, we searched for the shift that maximizes their spatial cross-correlation. For pairs that share a pattern, 92–98% of the cases result in a zero shift; that is, the domain gradients are already aligned, and phase alignment would leave them unchanged. We also measured the local spatial correlation between the domain gradients gsg^s, which is 0.08–0.41, compared with −0.01 to 0.08 for random pairs. This moderate value reflects domain-specific components, such as the opposite signs between MNIST and the color domains, rather than misalignment, since the best shift is already zero.

AuthorsOpenReview ↗

Response to Reviewer yFyW (3)

W3. Cross-architecture evaluation

We followed the requested procedure: each synthetic set is distilled once with ConvNet, and unseen architectures are then trained from scratch on that set. We report Digits-DG at IPC=10, averaged over four targets.

ConvNet (distillation)VGG-19ResNet-50
DM65.1 ± 1.038.2 ± 3.929.7 ± 1.5
DM + SGS67.7 ± 0.638.6 ± 1.538.8 ± 0.9
Gain+2.6+0.4+9.1

SGS improves DM on ConvNet and ResNet-50, with gains of +2.6 and +9.1, respectively. On VGG-19, SGS is comparable to DM, as the gain of +0.4 is within the standard deviation.

We added these results in Section 4.4 (Table 11).

We are glad to provide further detail on any of these points.

Reviewer_2oveOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies an interesting idea on domain-generalisable dataset distillation, where compact synthetic datasets are used to train models for unseen target domains. It proposes Spectral Gradient Surgery (SGS), which computes domain-wise gradients with respect to synthetic images and measures their agreement in the Fourier domain. SGS supplements the standard distribution-matching update with a consensus component aim to capture transferable information and a domain-specific component aim to preserve diversity. Extensive experiments on datasets such as Digits-DG, PACS, and CORe50-Hard demonstrate improvements over two distribution-matching methods, with additional offline distillation cost but no change to downstream training.

Critical strengths

This paper addresses a practically relevant limitation of dataset distillation, which is that synthetic datasets that preserve training utility within source domains may generalise poorly to unseen domains. The proposed method SGS consistently improves average OOD accuracy across three benchmarks, two synthetic-data budgets, and two distribution-matching methods.

The paper also includes useful control experiments to examine different factors such as the update magnitude, the contributions of individual gradient components, and pixel-space versus spectral agreement. These findings included in the analyses provide evidence that the improvement is not solely attributable to larger optimisation steps. The paper also reports its additional distillation cost and explores an extension to single-source generalisation.

Critical weaknesses

The interpretation of spectral agreement as transferable class information is insufficiently established. For exmaple, the agreement score is magnitude-weighted and can remain high when one domain dominates, even if other domains have opposing phases. As a result, the selected components may not need to reflect agreement across domains. Although Appendix A acknowledges that shared nuisance information can also be retained, the experiments do not establish when the proposed score reliably identifies transferable information. This weakens the evidence supporting the proposed mechanism by which SGS improves domain generalisation.

The contribution of the spectral operation is not sufficiently isolated across the main benchmarks. By linearity of the Fourier transform, the domain-specific component is simply the difference between a domain gradient and the mean gradient, requiring no spectral processing. Appendix D shows that adding this component alone already improves accuracy, but the comparison with full SGS and pixel-space agreement is limited to Digits-DG at IPC=10. Thus, it remains unclear whether the spectral construction provides consistent benefits beyond simpler domain-aware gradient mixing. The limited compute-matched evaluation further restricts assessment of the additional distillation cost.

Questions for authors

How sensitive is the agreement score to differences in gradient magnitude across domains? For example, coefficients of (100,-1,-1) yield agreement close to one despite opposing phases. Do the observed high-agreement components reflect broadly shared directions or dominance by individual domains?

Can the domain-mixing-only and pixel-space controls in Appendix D be evaluated on PACS and CORe50-Hard under comparable computational budgets? These comparisons would materially clarify whether the spectral operation is responsible for the improvements across the principal benchmarks.

How is the domain-component cosine similarity in Table 7 aggregated across domains and synthetic images? Since the residuals sum to zero across domains for each synthetic image, could the reported zero similarities arise from cancellation rather than orthogonality to the target gradient?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 2ove (1)

We thank the reviewer for the meaningful and valuable comments.

W1 / Q1. Sensitivity of the agreement score to gradient magnitude

We acknowledge that SGS cannot isolate semantic class information but instead identifies information shared across the source domains. Thus, we have replaced "class-relevant" with "source-consistent" throughout the manuscript. However, note that this limitation is common to other invariance-based methods, such as IRM, Fish, and Fishr. Nevertheless, the source-consistent component is empirically better aligned with unseen domains: gclassg^{\text{class}} is closer to the held-out target gradient than gg on every PACS target (cosine 0.27 vs. 0.21 on average; Table 9).

Although the example (100, −1, −1) is extreme, the analysis is correct: Eq. 8 can reward a single dominant domain. To assess the practical significance of this limitation, we evaluate the sensitivity of SGS to such dominance using real DM gradients at IPC=10 at the beginning of distillation, reporting the ranges across the four target domains of each dataset.

Digits-DGPACSCORe50-Hard
Domain grad norm ratio1.3–1.71.3–1.71.2–1.3
Actual consensus91–97%95–98%92–94%

Domain grad norm ratio denotes the median over classes of the max/min source-domain gradient norm, and Actual consensus denotes, among the frequency components with r>0.8r>0.8, the percentage in which every domain is within 90° of the mean phase. These results suggest that:

  • Dominance by one domain is rare: The gradient norms of the source domains are similar, with a median max/min ratio below 2.
  • High-agreement components reflect broadly shared directions: Over 90% of the components with r>0.8r>0.8 exhibit agreement across all domains.

Nevertheless, the magnitude can be helpful for weighting the update: with unit-length phase vectors, a low-magnitude coefficient that contributes little to the update receives the same vote as a dominant one, so components that should be suppressed can be emphasized. Accordingly, an equal-vote score still improves DM but performs slightly worse than rr (Section 4.3, Table 5).

We discussed these diagnostics in Section 4.4 (Table 8).

AuthorsOpenReview ↗

Response to Reviewer 2ove (2)

W2 / Q2. Contribution of spectral operation

We agree that Gs−GˉG^s - \bar G is the Fourier transform of gs−gˉg^s - \bar g, so the domain-specific component requires no spectral processing. However, the consensus weight rr is nonlinear, so measuring agreement in the frequency domain gives a different update from measuring it in pixel space, with two key differences. First, a real-valued pixel gradient can only point in one of two directions (sign), while the phases give a continuous angle between domains, which provides a finer-grained measure of agreement. Second, each Fourier coefficient corresponds to a global pattern over the image, so one agreement decision keeps or attenuates one spatially coherent pattern, while pixel agreement decides each pixel independently, which may hinder spatial consistency.

Additional Results on PACS and CORe50-Hard. To isolate the contribution of the Fourier operation, we evaluate the domain-mixing-only control and the pixel-agreement control of Appendix D on PACS and CORe50-Hard at IPC=10, under comparable computational budgets. Both controls use the same domain gradients as SGS, so their distillation cost is the same (the cost of the Fourier transform is negligible). We also run DM for 66k iterations (3.3×), which matches the wall-clock cost of SGS. DM and SGS (20k iter.) are from Table 1.

PACSCostPACSAvg
DM (20k iter.)1.0×48.027.933.832.635.6
SGS (20k iter.)3.25×53.828.435.137.238.6
DM (66k iter.)3.3×48.427.132.232.535.0
Domain mixing only (20k iter.)3.25×51.325.032.932.335.4
Pixel agreement (20k iter.)3.25×47.325.934.533.535.3
CORe50-HardCosts5s7s9s10Avg
DM (20k iter.)1.0×28.431.430.319.327.4
SGS (20k iter.)3.25×29.832.632.321.028.9
DM (66k iter.)3.3×29.932.729.917.427.5
Domain mixing only (20k iter.)3.25×28.529.727.419.226.2
Pixel agreement (20k iter.)3.25×29.531.828.419.827.4

These results suggest that:

  • The gain of SGS is not explained by additional compute: Running DM 3.3× longer barely changes accuracy on either benchmark.
  • The spectral agreement contributes beyond domain mixing and pixel agreement: Neither control recovers the gain of SGS on either PACS or CORe50-Hard.

We added these results to Appendix D (Table 16).

AuthorsOpenReview ↗

Response to Reviewer 2ove (3)

Q3. Aggregation in Table 7

We acknowledge the reviewer's concern. The reported value was the signed cosine averaged over the three sources and all synthetic images of all classes. Since the residuals sum to zero, this average largely cancels. Therefore, the near-zero entries did not indicate orthogonality. As reported below, when evaluated per source, the residuals are not orthogonal to the held-out target gradient.

Target (sources 1/2/3)Source 1Source 2Source 3
Digits M (N/V/Y)+0.64−0.62−0.25
Digits N (M/V/Y)+0.18−0.20−0.11
Digits V (M/N/Y)−0.61+0.03+0.67
Digits Y (M/N/V)−0.49−0.04+0.51
PACS P (A/C/S)+0.57+0.08−0.47
PACS A (P/C/S)+0.54+0.12−0.51
PACS C (P/A/S)−0.02+0.12−0.04
PACS S (P/A/C)−0.27−0.12+0.27
CORe50-H s5 (s7/s9/s10)−0.02−0.03+0.03
CORe50-H s7 (s5/s9/s10)−0.20+0.15+0.05
CORe50-H s9 (s5/s7/s10)−0.10+0.28−0.16
CORe50-H s10 (s5/s7/s9)−0.02+0.18−0.14

Each entry denotes the cosine similarity between the residual gdomaing^{\text{domain}} of one source and the gradient of the held-out target. For example, with M as the target, the residual of N has a similarity of +0.64 and that of V has −0.62. We observe that the signs differ across sources on every target. The residual of a source close to the target points toward it, while that of a distant source points away. This aligns with the visual similarity between domains: M and N in Digits-DG, and P and A in PACS, are highly similar. This also suggests the role of gdomaing^{\text{domain}}: assigning the source domains uniformly across the synthetic samples diversifies the synthetic set, which may help cover unseen domains.

We have revised the discussion and Table 7 (now Table 9) in Section 4.4 ("Alignment with Unseen Domains"), and added the full table above as Table 17 (Appendix D.1).

We are glad to provide further detail on any of these points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.