acceptodds
Under review as a conference paper at ICLR 2027

FedSpec: Shared Spectral Structures for Heterogeneous Federated Learning

Abstract

Model heterogeneity poses a fundamental challenge to federated learning, as different model architectures produce incompatible parameter and representation spaces. Existing heterogeneous federated learning methods mainly rely on parameter sharing or feature alignment, often requiring compatible model components or explicit alignment mechanisms. To address this issue, we propose FedSpec, a heterogeneous Federated learning framework based on shared Spectral structures. FedSpec transforms heterogeneous local feature spaces into fixed-length normalized spectral structures through singular value decomposition, aggregates them into a global spectral structure, and uses the global spectrum as a structural prior for local spectral correction. The singular values characterize the relative importance distribution of principal representation directions, enabling a unified structural representation across heterogeneous models without requiring parameter compatibility or explicit feature alignment. We further analyze the convergence behavior and information reduction properties of FedSpec. Extensive experiments across diverse architectures and data modalities demonstrate that FedSpec consistently outperforms state-of-the-art methods, achieving up to 7.67% accuracy improvement with substantially lower communication cost.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Heterogeneous Federated Learning, Shared Spectral Structure

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pXHsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper highlights an important challenge in HFL: conventional model sharing and feature alignment, which require consistent feature-layer dimensions, may not work well across heterogeneous client architectures.

  • To address this, the paper proposes FedSpec, a novel three-stage HFL algorithm that leverages SVD to enable knowledge sharing without requiring dimensional consistency.

  • Comprehensive experiments show that FedSpec achieves strong performance with improved communication efficiency.

Critical strengths

The paper proposes a simple yet effective approach to HFL that applies SVD to share knowledge across heterogeneous architectures. It resolves the dimensional-consistency limitation with minimal client-side modifications, making it practical and easy to adopt.

Critical weaknesses
  • The reviewer's main concern is technical novelty. Although the paper applies SVD to representations in HFL, SVD is widely used in both centralized and federated learning, e.g., for analyzing representation compactness (Choi et al., Statistical Characteristics of Deep Representations: An Empirical Investigation) and for SVD-based compression of model updates in FL (Wu et al., Communication-Efficient Federated Learning via Knowledge Distillation). While the paper applies SVD in a well-structured way, the contribution would be stronger with methodological novelty.

  • While the experiments are comprehensive, most baselines are not designed around communicating representations, so the comparison does not clearly show FedSpec’s advantage over its relevant competitors, such as FedTGP (Zhang et al.) and FedSA (Zhou et al.).

Questions for authors

Q1. Relaxing feature-dimension consistency is useful, but its practical impact seems limited. The paper also implies that explicit feature-space alignment is problematic. Please clarify these points.

Q2. Some statements are strong or qualitative without sufficient support (e.g., lines 311–313), and some of the described properties of the spectral structure appear to overlap (lines 245–252). Please clarify these points.

Q3. In Figure 4, performance appears largely insensitive to m. Could the authors explain this quantitatively, e.g., by reporting how much spectral energy the top-m components capture?

Q4. Performance on CIFAR-100 varies noticeably with m, λ, and w. Could the authors briefly suggest a practical way to tune these hyperparameters in realistic FL settings?

1 comment
AuthorsOpenReview ↗

Thank you for recognizing the novelty of FedSpec, its simple yet effective design for heterogeneous federated learning, its ability to relax feature-dimension consistency with minimal client-side modifications, and its comprehensive empirical validation with strong performance and communication efficiency.

W1: We agree that SVD itself is a well-established mathematical tool; the novelty of FedSpec lies in introducing a new collaboration interface for heterogeneous models. Choi et al. mainly use rank-related statistical properties to analyze and regularize deep representations, while FedKD applies SVD for low-rank compression of communicated gradients. In contrast, FedSpec maps local feature spaces with different architectures and representation dimensions into fixed-length normalized spectral structures that can be directly aggregated at the server and then fed back as a global structural prior for local optimization. Based on this interface, FedSpec further integrates spectral abstraction, spectral aggregation, and spectral correction into a complete collaboration mechanism, enabling knowledge sharing across heterogeneous models without parameter compatibility or explicit feature alignment. We will further emphasize this methodological distinction in the revision.

W2: Thank you for pointing out FedTGP and FedSA as relevant competitors. FedTGP and FedSA establish heterogeneous collaboration through class-wise prototypes or semantic anchors, whereas FedSpec uses a class-agnostic spectral structure as the collaboration interface. We additionally compare them under all six settings in Table 1. For FedTGP, the accuracies are 50.84%/35.61%, 57.92%/36.54%, and 59.83%/40.17% on CIFAR-100/Tiny-ImageNet at 20%, 10%, and 5% participation, respectively; for FedSA, the corresponding results are 51.63%/35.96%, 58.91%/36.88%, and 61.12%/41.06%. FedSpec achieves 61.94%/43.10%, 63.70%/45.09%, and 64.56%/47.30%, outperforming FedTGP by 4.73%–11.10% and FedSA by 3.44%–10.31% across all six settings. These comparisons further demonstrate the advantage of spectral-structure-based collaboration over closely related prototype/semantic-based heterogeneous FL methods.

Q1: FedSpec requires neither explicit feature alignment nor consistent representation dimensions. For client ii, SRS/GRA constructs Ri∈Rm×diR_i\in\mathbb{R}^{m\times d_i}, whose SVD gives Ui∈Rm×mU_i\in\mathbb{R}^{m\times m}, Σi∈Rm×m\Sigma_i\in\mathbb{R}^{m\times m}, and Vi∈Rdi×mV_i\in\mathbb{R}^{d_i\times m}. FedSpec only extracts and normalizes the mm singular values in Σi\Sigma_i into si∈Rms_i\in\mathbb{R}^{m}. Thus, different did_i are naturally mapped to the same mm-dimensional spectral structure without dimension matching or feature-wise alignment.

Q2: For Lines 311–313, GRA averages representations within groups from the current epoch, so each group representation integrates information from multiple samples and is therefore less sensitive to sample-level fluctuations than direct sampling. We will replace “more accurate” with “more stable/robust” to avoid an overly strong statement. For Lines 245–252, the earlier three properties are general design requirements for an effective structural abstraction, while this part explains how the spectral structure in FedSpec specifically realizes these requirements; thus, it is a concrete instantiation rather than a repetition.

Q3: We believe the reviewer refers to Appendix Figure 5. In FedSpec, mm is not a top-mm truncation of a fixed spectrum; instead, it determines both the number of representative features used to construct Ri∈Rm×diR_i\in\mathbb{R}^{m\times d_i} and the granularity of the resulting spectral structure. To further quantify the robustness to mm, we analyze the cumulative spectral energy E(q)=∑k=1qσk2/∑kσk2E(q)=\sum_{k=1}^{q}\sigma_k^2/\sum_k\sigma_k^2. On CIFAR-100, the top-10/30/50 components capture 88.9%/96.1%/98.2% of the spectral energy; on Tiny-ImageNet, the corresponding values are 68.7%/84.9%/91.8%, and the top-100 components capture 97.6%. These results show that the dominant structural information is concentrated in the leading spectral components, so increasing mm further introduces only limited additional information, consistent with the relatively stable performance over a broad range of mm in Appendix Figure 5.

Q4: In fact, mm, λ\lambda, and KK have only limited impact on FedSpec. On CIFAR-100, the narrow y-axis visually amplifies small performance variations; Tiny-ImageNet is more challenging and therefore shows slightly larger fluctuations, but the overall performance remains stable. In practice, mm can be selected based on spectral energy and communication/computation budgets, λ\lambda can be coarsely searched within a small-to-moderate range, and KK can prioritize the later local epochs. Therefore, FedSpec does not require fine-grained hyperparameter tuning.

Reviewer_tD9QOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedSpec, a heterogeneous federated learning framework based on shared spectral structures. Instead of sharing model parameters or aligning feature representations, FedSpec transforms each client’s local representation space into a fixed-length normalized spectral vector via SVD of a centered representation matrix. These local spectra are aggregated by the server into a global spectral structure, which is then broadcast and used as a structural prior for local spectral correction. The method is architecture-independent and dimension-independent in the sense that clients communicate only an m-dimensional normalized singular-value vector. The paper provides convergence and information-reduction analysis, and reports experiments on CIFAR-100, Tiny-ImageNet, Fed-Heart-Disease, and Stack Overflow . FedSpec consistently outperforms several HFL baselines and shows lower communication cost than FedMRL.

Critical strengths

1.A simple and architecture-independent collaboration interface. The use of normalized singular values as a shared structural abstraction is novel and practically appealing. This is a conceptually appealing way to facilitate collaboration among models with different representation dimensions. The method is relatively simple to integrate into heterogeneous local models, and its communication payload does not scale directly with the number of model parameters.

2.Broad empirical validation. The method consistently outperforms the reported baselines across the two main image-classification benchmarks and different client participation rates. The paper also includes experiments involving more diverse architectures and data modalities, together with sensitivity analyses and ablations of the main design components. These results provide empirical support for the practical potential of the proposed approach.

Critical weaknesses
  1. The communication-efficiency and privacy claims need more careful qualification.The paper reports very large communication-cost reductions relative to FedMRL, but the comparison would be easier to assess if the accounting explicitly covered all transmitted information and used a consistent definition of communication cost across methods.

  2. The information-reduction results do not establish resistance to privacy attacks. Although communicating normalized spectra discards singular vectors and absolute scale, the remaining spectrum may still reveal information about local data distributions or learned representations. The privacy-related conclusions should therefore be framed as information reduction rather than as a privacy guarantee.

Questions for authors
  1. Can the authors clarify precisely what is guaranteed by Theorem 1? In particular, can the bounded inter-round variation assumption be justified or empirically assessed for the proposed spectral aggregation procedure? Does the analysis extend to a global objective across heterogeneous clients, or is the guarantee intentionally limited to each client's time-varying composite objective?

  2. The information-reduction analysis is not a formal privacy guarantee. Could you provide an empirical leakage or reconstruction study to quantify what information about local data or representations may still be inferable from the uploaded spectrum?

  3. How should practitioners choose the spectral dimension m and the correction coefficient λ? Are there theoretical or empirical guidelines beyond the default values used in the experiments?‘

1 comment
AuthorsOpenReview ↗

Thank you for recognizing the novelty and practical appeal of using normalized spectral structures as an architecture-independent collaboration interface, the low communication payload that is independent of model parameter size, and the broad empirical validation and strong performance of FedSpec across diverse architectures and data modalities.

W1: We use a consistent communication accounting for both methods. In FedSpec, each client uploads its local (m)-dimensional spectral structure and receives the aggregated global (m)-dimensional spectrum; similarly, in FedMRL, clients upload and receive the shared homogeneous-model parameters. Thus, the uplink and downlink payloads are symmetric within each method. Fig. 3 compares the cumulative communication cost required to reach the same target accuracy under the same accounting rule, and FedSpec benefits from both a substantially smaller per-round payload and fewer communication rounds. Since the downlink has the same order of cost as the uplink, including both directions does not change the relative communication advantage between FedSpec and FedMRL. We will clarify this accounting scope in the revision.

W2: We agree that information reduction should be clearly distinguished from a formal privacy guarantee, which is also the intended scope of our analysis. Appendix C.2 explicitly states that FedSpec does not provide a formal privacy guarantee. Theorem 2 only shows that spectral abstraction does not increase mutual information with the local data compared with directly communicating the representation matrix, while Theorem 3 establishes that the normalized spectral mapping is non-injective and therefore does not uniquely determine the underlying representation. Thus, our claim is limited to information reduction, rather than resistance to specific privacy attacks. We will further unify the wording in the revision to avoid any possible ambiguity.

Q1: Theorem 1 is intentionally a per-client convergence result for the time-varying composite objective Jit=Li+LspectJ_i^t=L_i+L_{\mathrm{spec}}^t, explicitly accounting for the bias introduced by the two-stage update. Under Assumptions 1–4, it guarantees an O(1/TE)O(1/\sqrt{TE}) average first-order stationarity rate, where TT denotes the active communication rounds of client ii; it does not claim convergence to a single global objective shared by heterogeneous clients. Assumption 4 controls the objective drift induced by the evolving global spectrum. Empirically, Figs. 8–9 show that the local-global spectral discrepancy decreases overall and becomes increasingly stable during training, providing empirical support for this bounded-variation assumption. We will clarify this scope more explicitly in the revision.

Q2: Following the reviewer’s suggestion, we additionally conduct a white-box reconstruction attack, where the attacker is given access to the local feature extractor and attempts to reconstruct local samples from the communicated information. On CIFAR-100, reconstruction from raw representations achieves PSNR/SSIM/LPIPS of 21.6/0.54/0.34, whereas reconstruction from FedSpec spectra yields 9.4/0.11/0.76. On Tiny-ImageNet, the corresponding results are 18.3/0.42/0.43 versus 8.1/0.07/0.82. These results show that reconstruction from the uploaded spectrum is substantially less effective than from raw representations, demonstrating that FedSpec effectively reduces reconstruction leakage and provides strong empirical privacy protection under the evaluated attack setting.

Q3: The spectral dimension mm mainly controls the granularity of the shared spectral structure together with its computational and communication cost, so in practice a relatively small mm can be selected to balance structural expressiveness and efficiency. The coefficient λ\lambda controls the strength of the global spectral prior and can generally be chosen in a small-to-moderate range. As shown in Appendix Figs. 5–6, FedSpec remains stable over a broad range of both mm and λ\lambda and consistently outperforms FedMRL, indicating that these parameters do not require fine-grained tuning. We will add this practical guideline in the revision.

Reviewer_eL9GOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed FedSpec, a heterogeneous FL framework based on shared spectral structures. Unlike methods that share model parameters or explicit feature alignment, FedSpec introduces shared spectral structures as a new collaboration interface for heterogeneous FL. The server constructs a global spectral structure through weighted aggregation, and each client uses the global spectrum as a structural prior for local spectral correction. The convergence behavior and information reduction properties are theoretically analyzed.

Critical strengths
  1. The designed collaboration interface offers good architectural compatibility, avoiding the direct aggregation of heterogeneous model parameters.
  2. The advantage in communication efficiency is clear.
  3. The experimental results are competitive.
Critical weaknesses
  1. The author presented that the aggregated global spectral structure captures common structural patterns across clients. However, in high non-IID scenarios, the principal subspaces learned by different models may be completely misaligned semantically. How to guarantee the effectiveness of FesSpec in this setting.
  2. In Table 1, except FedSpec, the accuracy of other baselines is all lower than that of Standalone (55.37%) on CIFAR-100 (20% participation rate). Please clarify this phenomenon.
  3. In Table 1, with the participation rate decreasing (20%-10%-5%), the accuracy of FedSpec on CIFAR-100 is increasing, please explain whether this caused by specific settings.
Questions for authors

FedSpec is a novel heterogeneous FL framework based on shared spectral structures. It abstracts heterogeneous local feature spaces into compact spectral structures, aggregates them into a global spectral structure, and uses the global structure as a structural prior to guide local representation optimization. The experimental evidence is through. Although the weaknesses pointed out are worth noting, they are not fundamental flaws, but rather opportunities for further improvement. The recommendation is “Weak Accept” as its core contributions are novel. However, addressing the following points in the rebuttal would strengthen the final paper considerably, 1) the effectiveness of FedSpec in highly non-IID scenarios, and explain the principle behind it. 2) other points in Weaknesses.

1 comment
AuthorsOpenReview ↗

Thank you for recognizing the architectural compatibility of FedSpec, its clear communication-efficiency advantage, competitive experimental performance, and the novelty of its core contributions.

  1. FedSpec does not require the principal subspaces learned by different clients to be semantically aligned, because what we share and aggregate are not singular vectors or specific principal directions, but the sorted and normalized singular values that characterize the relative importance distribution across principal directions. Therefore, even if different clients learn principal directions with different semantic meanings, this does not directly affect the aggregation of their spectral structures. Theoretically, Theorem 3 shows that different representation matrices with different principal directions can yield the same spectral structure, indicating that FedSpec does not rely on semantic consistency of principal directions across clients. Empirically, the main experiments already adopt severe label-skew settings, and Appendix Fig. 4 further covers different levels of pathological non-IID and Dirichlet-based non-IID, where FedSpec consistently maintains stable performance under strong data heterogeneity.

  2. Standalone is not necessarily a lower bound for heterogeneous FL methods. Under severe data and model heterogeneity, transferred knowledge may conflict with local representations or optimization objectives, leading to negative transfer and thus lower performance than independent local training. This phenomenon mainly appears in the CIFAR-100 setting with 20% participation; under other participation rates, some baselines already outperform Standalone, indicating that it is not a systematic failure of the baselines. In contrast, FedSpec communicates only compact spectral structures while preserving each client's own parameter and representation spaces, which helps maintain effective collaboration in this setting.

  3. This phenomenon is not caused by participation-specific parameter tuning. Across all participation rates, we keep the data partition, (T=500), (E=10), optimizer, and major FedSpec hyperparameters unchanged, and vary only ρ\rho. Notably, the same trend is observed on both CIFAR-100 and Tiny-ImageNet, suggesting that it is not a dataset-specific anomaly. Under strong data and model heterogeneity, a higher participation rate aggregates more diverse local spectral structures within the same round, which can increase interference among heterogeneous structural information. A lower participation rate can alleviate such within-round interference, while long-term random participation still maintains cross-client knowledge exchange, leading to higher final performance under the evaluated settings.

Comments on the paper

2 comments

Dear reviewers,

The authors have responded to your concerns. Please review their responses as soon as possible and provide any necessary feedback. Thank you.

AC

Dear Reviewers,

Thank you again for your constructive feedback. We have carefully addressed all concerns and questions in our rebuttal, including additional analyses and experiments where requested. As the discussion period is ongoing, we would greatly appreciate it if you could take a moment to review our responses and let us know whether your concerns have been adequately addressed. We are also happy to provide any further clarification if needed.

Thank you very much for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.