acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Heterogeneous Federated Learning: Low-Rank Continuous Feature Representations for Knowledge Sharing

Abstract

Real-world federated learning deployments are often model-heterogeneous, rendering parameter averaging ineffective. Existing methods mainly adopt two mechanisms: one introduces homogeneous proxy models for knowledge aggregation and distillation, but incurs high computation and communication costs; the other shares class prototypes, which are lightweight but compress intra-class structure and are prone to feature collapse and semantic mismatch. Therefore, existing methods exhibit a clear expressiveness–efficiency gap. We propose FedCF, which leverages lightweight yet expressive low-rank continuous feature fields to support heterogeneous collaboration, preventing feature collapse while maintaining low overhead. The server aligns and fuses heterogeneous local fields in a canonical function space to form a queryable global field, providing clients with rich feature supervision to mitigate collapse. Local training further adopts feature–logit dual distillation to jointly align representations and calibrate decision boundaries, without proxy models or public data, keeping the overhead low. Across heterogeneous benchmarks, FedCF achieves a better accuracy–efficiency trade-off.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Heterogeneous Federated Learning, Continuous Feature Representation, Low-Rank

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6G9kOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedCF, a model-heterogeneous federated learning framework based on low-rank continuous feature fields. It represents class-conditional spatial features through compact continuous functions and aligns heterogeneous representations in a shared canonical space. Feature and logit distillation further facilitate knowledge transfer across clients. Experiments on three image classification datasets demonstrate improvements in accuracy and efficiency.

Critical strengths
  1. The paper addresses an important challenge in heterogeneous federated learning by exploring a more expressive alternative to discrete prototypes without relying on auxiliary models.

  2. The low-rank continuous feature representation offers an interesting approach to cross-architecture knowledge sharing. The proposed framework also considers both representation alignment and classifier calibration.

Critical weaknesses
  1. The central claim of preserving rich intra-class structure is insufficiently justified, as the continuous fields primarily capture class-conditional mean features and may still lose sample-specific variations.

  2. The experiments are limited to 30 fully participating clients, while communication costs remain substantially higher than prototype-based baselines. This weakens the claimed efficiency advantage and practical applicability.

Questions for authors
  1. How do continuous feature fields preserve intra-class variations beyond spatially varying class means?

  2. How does Eq. (5) prevent semantic misalignment across heterogeneous architectures?

  3. Does Theorem 2.8 provide convergence guarantees for the complete federated training process?

  4. How would FedCF perform against prototype-based methods under matched communication budgets?

8 comments
AuthorsOpenReview ↗

Q1: Intra-Class Structure

Q1. Preservation of Intra-Class Structure and Sample-Specific Variations

We thank the reviewer for the insightful comments. FedCF uses continuous feature fields to preserve feature differences across spatial locations within the same class, while allowing clients to independently learn features from real images. We clarify the mechanism and provide direct experimental evidence below.

1. Mechanism for Preserving Intra-Class Structure

Under Assumption 2.2, Eq. (3) learns the expected class features at different spatial locations. Conventional class prototypes compress an entire class into a single vector, ignoring spatial feature differences. In contrast, the shared learnable coordinate functions and low-rank cores in Eq. (2) model both location-dependent features and their relationships, preserving richer class-level spatial structure and supporting continuous queries at different resolutions. Corollary 2.6 establishes that a class prototype is a spatially constant special case of a continuous field. Lemma 2.7 further proves that the continuous field has no greater optimal approximation error than a single class prototype.

Regarding sample-specific variations, the continuous field provides shared class-level spatial knowledge rather than replacing features extracted from actual images. Even when images of the same class share the same field target at a given location, each client still extracts features from its own input. Eq. (7) provides feature-alignment supervision, while Eq. (9) retains the ground-truth classification loss and prediction distillation; these objectives do not require features from different samples to be identical. Thus, FedCF facilitates collaboration through shared spatial regularities while preserving sample-specific feature learning from real inputs.

2. Verification on a Held-Out Test Set

Fig. 3 in the original paper illustrates differences in spatial responses across samples of the same class, while Fig. 4 demonstrates the reconstruction advantage of continuous fields. We further analyze spatial structure, sample residuals, and reconstruction errors on the held-out CIFAR-100 test set. Spatial mean variance measures variation in class-mean features across locations; sample residual variance measures feature differences between same-class samples at the same location relative to the class mean; and real test-feature MSE evaluates how accurately prototypes and continuous fields reconstruct actual features of unseen samples.

Table R9: Held-out reconstruction and variance analysis

Feature spaceSpatial mean varianceSample residual varianceReal test-feature MSE ↓ (Prototype / FedCF)
Local feature space0.1600.0900.264 / 0.175
Global canonical space0.0320.0100.046 / 0.027

The results show that: a. Spatial structure is effectively modeled. Nonzero spatial mean variance in both spaces confirms location-dependent intra-class structure that a single prototype cannot capture. FedCF also achieves lower reconstruction error for unseen test features in both spaces, demonstrating a more accurate representation of spatial features. b. Sample-specific feature differences remain. Sample residual variances of 0.090 and 0.010 in the local and global canonical spaces, respectively, show that features extracted from different images of the same class have not all collapsed to an identical class mean.

These findings are consistent with the classification gains in Table 1, on CIFAR-100, FedCF achieves 59.30% accuracy, outperforming the strongest prototype baseline (55.63%). In summary, continuous low-rank functions enable FedCF to more accurately represent and share spatial feature structure that a single class prototype cannot retain, while clients continue learning features from individual real images. This improves the expressiveness of shared knowledge without forcing all same-class samples' actual features into a single class vector.

AuthorsOpenReview ↗

Q2: Cross-Architecture Semantic Alignment

Q2. Cross-Architecture Semantic Alignment in Eq. (5)

We thank the reviewer for the question. FedCF establishes semantic correspondence across heterogeneous architectures in a shared canonical space through semantically meaningful local features, class-conditioned global fields, and feature and coordinate Translators.

1. Mechanism of Cross-Architecture Semantic Alignment

First, alignment in FedCF is grounded in class semantics. Clients learn discriminative backbone features under ground-truth label supervision, and Eq. (3) separately fits a local continuous field for each class, providing class-specific feature references for subsequent alignment. Second, architectures differ in feature dimensions, spatial resolutions, and feature distributions. FedCF uses the feature mapping AiA_i and coordinate mapping ψi\psi_i to transform their features and spatial locations, respectively, into a shared canonical space. Eq. (5) then requires same-class features from different architectures to jointly fit one shared global class field, while different classes have separate global fields. Thus, alignment is not merely about making feature values from different architectures numerically similar; it encourages consistent representations for the same class at corresponding spatial locations.

Specifically, Eq. (5) uses alternating optimization: with the Translators fixed, the global fields aggregate same-class features from different architectures; with the global fields fixed, the Translators adjust their mappings toward the shared target for each class. When the fixed class targets are distinguishable, incorrect class mappings increase alignment error, providing class-dependent optimization gradients. Furthermore, the learned canonical-space knowledge supervises features extracted from real client images through Eq. (7), together with the label-based classification supervision in Eq. (9), promoting continued semantic alignment across architectures.

2. Direct Verification of Cross-Architecture Semantic Consistency

We add cross-architecture semantic diagnostics and classification tests on real CIFAR-100 test images, with reference values:

Table R10: Cross-architecture semantic consistency and class separability

DiagnosticReferenceFedCF
Cross-architecture semantic gap ↑≈0 (random class correspondence)0.25
Inter-class cosine similarity of global fields ↓1 (complete class collapse)0.62
Classification accuracy on real test features ↑1% (random guessing)42.0%

The cross-architecture semantic gap is the difference between the mean same-class and different-class cosine similarities of mapped real test features from different architectures. FedCF achieves 0.25, showing that same-class features from different architectures are more similar than different-class features, indicating meaningful semantic correspondence. The inter-class cosine similarity of the global fields is 0.62, further indicating that different classes remain distinguishable in the canonical space.

We also pass independent test images through the client backbones and feature Translators, then feed the mapped features directly into the frozen global classifier. This inference procedure uses neither ground-truth class labels nor class-conditioned queries to the global fields. FedCF achieves 42.0% accuracy, far above the 1% random-guessing baseline. This demonstrates that class semantics are retained not only in the global fields but also in features extracted from real images and mapped from different architectures.

In summary, FedCF promotes cross-architecture semantic alignment through shared class-conditioned targets and alternating optimization. We have clarified the class-conditional alignment mechanism in Sec. 2.2.2 and added canonical-space diagnostics in Tables 2 and 12 of the revised manuscript.

AuthorsOpenReview ↗

Q3: Scope of Theorem 2.8

Q3. Scope of the Convergence Guarantee in Theorem 2.8

We thank the reviewer for the careful examination. Theorem 2.8 provides a conditional nonconvex stationarity guarantee for FedCF's client-side optimization under dynamically updated global-teacher supervision. It also explains how multi-point spatial supervision and global-teacher quality affect client optimization.

1. Theoretical Guarantee of Theorem 2.8

During FedCF training, clients optimize against a fixed global teacher within each communication round, while the teacher is updated across rounds. Thus, the analysis covers client optimization over multiple rounds rather than a single round. Specifically, Eq. (10) bounds the average gradient norm of the ideal client objective and reveals three key properties:

(a) The initial optimization-error term decreases with training. As the number of client iterations increases, the contribution of the initial optimization gap to the bound diminishes.

(b) Multi-point queries help stabilize training. When spatial query points are independently sampled, using more points reduces gradient-estimation variance, providing theoretical support for multi-point spatial supervision.

(c) Global-teacher quality affects optimization accuracy. The gradient discrepancy between the actual and ideal teachers contributes to the residual error in the stationarity bound, showing why more accurate global knowledge can improve client optimization.

Under the stated conditions, Theorem 2.8 therefore provides an explicit upper bound on the average stationarity measure of the ideal local objective.

2. Full Teacher Error and Scope of the Analysis

FedCF's actual teacher consists of the global continuous field, feature and coordinate Translators, and global classifier. Appendix E.4 already defines the full gradient discrepancy between the actual and ideal teachers in Eq. (34), and derives the nonconvex stationarity bound incorporating this discrepancy in Eqs. (36)–(47). To align the theorem statement precisely with the actual training process, we have revised the teacher-error term in Eq. (10), following Eq. (47), to be the average squared norm of the full teacher-gradient discrepancy over all local training steps. This accounts for the combined effects of all teacher components without introducing new assumptions or changing the algorithm or the main proof.

In summary, the value of Theorem 2.8 lies in its conditional nonconvex stationarity guarantee for client optimization across communication rounds and its explicit characterization of how multi-point queries reduce gradient variance and how teacher quality affects optimization accuracy. It provides direct theoretical support for guiding heterogeneous clients with continually updated global knowledge in FedCF.

Reviewer_wqXBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FedCF for federated learning with heterogeneous client architectures. Instead of exchanging model parameters or a single feature prototype per class, each client fits a class-conditional continuous feature field mapping normalized spatial coordinates to expected backbone features. The fields use a low-rank tensor-function parameterization. The server learns feature and coordinate translators together with global fields in a canonical space, then trains a global classifier on translated features queried from local fields. Clients use the resulting shared knowledge through feature alignment and logit distillation. The paper provides representation and conditional convergence analyses, and evaluates the method on CIFAR-10, CIFAR-100, and Tiny-ImageNet with heterogeneous CNN/ViT backbones. The reported results show improvements over prototype baselines and substantially lower communication than several proxy-model baselines.

Critical strengths
  1. The paper presents an interesting perspective on heterogeneous federated learning by exploring richer spatial representations for knowledge sharing beyond conventional class prototypes.
  2. The framework addresses a practically relevant challenge: balancing knowledge expressiveness and communication efficiency while enabling collaboration across different architectures without public data.
  3. The experiments across multiple datasets and heterogeneous settings provide encouraging evidence of the framework’s effectiveness, while the ablation studies help clarify the contributions of its main components.
Critical weaknesses
  1. Unresolved alignment collapse. Eq. (5) admits Ai(z)=Sc(s,t)=bA_i(z)=S_c(s,t)=b for a fixed unit vector bb, yielding zero loss without semantic information. Neither normalization nor alternating updates excludes this solution, and Eq. (6) updates only the classifier. What prevents collapse in the actual implementation?

  2. Conditional theoretical support. Theorem 2.8 assumes that teacher-gradient error is bounded by the best achievable local-field approximation error. This relationship is not established for the learned global fields, translators, and classifier. Please justify it or narrow the theoretical claims.

  3. Spatial means do not establish sample-level diversity. Eq. (3) estimates one class-conditional mean per location, so different same-class images receive the same spatial target. Please distinguish spatial structure from sample-level variation and provide held-out evidence supporting the diversity-preservation claim.

  4. Insufficiently controlled representation comparisons. The rank-1 ablation does not isolate continuity from additional spatial capacity. Please compare against discrete spatial means or multiple prototypes using identical translators, losses, and communication budgets. The CAM comparisons also use different scalar scores (Eqs. (70)–(71)), limiting their interpretation.

  5. Unclear distillation interpretation. With identical queried points and frozen teachers, averaging teacher-to-student KL losses and distilling the mean teacher distribution produce identical student gradients. Their similar accuracy in Table 5 therefore does not establish teacher consistency. Please clarify this interpretation and why repeated queries outperform a precomputed client/class soft target.

  6. Incomplete efficiency accounting. Appendix F.1 reports client-side computation without quantifying server fitting or end-to-end runtime. Compression relative to spatial grids also needs justification: for a 4×44\times4 grid, a rank-6 core contains 36di36d_i values versus 16di16d_i for the spatial means, excluding factor networks.

Questions for authors

See the weakness.

7 comments
AuthorsOpenReview ↗

W1: Alignment Collapse

W1. Alignment Collapse in Eq. (5)

We thank the reviewer for this concern. We clarify the source of class-dependent gradients in Eq. (5) and directly diagnose representation collapse in canonical space.

1. Optimization Mechanism

FedCF's global alignment has supervised class semantics: client backbones learn from true-label classification supervision, while Eq. (3) fits a local continuous field for each class, updated throughout local training. Thus, global alignment always has class-semantic references from real data.

Eq. (5) further uses class-conditional alternating optimization to suppress joint collapse. When updating the global field, the feature and coordinate Translators are fixed, and independent class-specific cores fit the mapped features of each class. When updating the Translators, the global field is fixed, and the shared feature and coordinate Translators match class-specific global targets. If the fixed references distinguish classes, mapping all classes to one constant incurs extra fitting error; the minimum MSE of constant outputs is no less than the reference-feature variance. Thus, Eq. (5) directly provides class-dependent gradients to the Translators through class-specific targets. Alternation also prevents the global field and Translators from co-adapting toward degeneration within the same update. These gradients arise from Eq. (5)'s alignment loss, not Eq. (6), which updates only classifier parameters.

2. Direct Experimental Evidence Against Class Collapse

Fig. 6(b) already shows that alternating optimization yields higher accuracy and lower alignment error than joint optimization. We further diagnose representation degeneration for both on CIFAR-100.

Table R3: Accuracy, alignment error, and representation degeneration

StrategyAcc. (%) ↑Align MSE ↓Collapse Ratio (≈1)Norm Ratio (≈1)
Joint optimization57.840.00450.6250.554
FedCF (alternating)59.300.00310.8490.878

Collapse Ratio and Norm Ratio measure how much the global field retains the variance and magnitude of locally mapped features, respectively; values closer to 1 mean better retention. Alternating optimization raises accuracy by 1.46 percentage points, reduces alignment error, and markedly improves both retention ratios. Thus, it improves not only fitting performance but also mitigates excessive contraction of feature differences and magnitudes. We further examine global-field class separability and semantic preservation in real test images after Translator mapping.

Table R4: Canonical-space class separability and semantic preservation

DiagnosticFully collapsed referenceFedCF
Global-field inter-class cosine similarity ↓10.62
Global-field class-center distance ↑00.86
Effective rank of centered class centers ↑029.6
Cross-architecture semantic gap ↑00.25
Classification accuracy on real test features (%) ↑≈1.042.0

The first three metrics show that classes do not collapse to an identical constant but retain multidimensional class differences. The cross-architecture semantic gap further shows that mapped features across architectures preserve class-related semantic relationships. We pass independent test images through client backbones and the feature Translator, then classify them directly with the frozen global classifier. Without ground-truth labels as input or class-conditioned global-field queries, accuracy reaches 42.0%, far above ≈1% random guessing. Thus, class-discriminative information is retained not only in class-conditioned global-field parameters but also in real image features mapped by the Translator. Overall, FedCF shows no class-indistinguishable representation collapse during training.

AuthorsOpenReview ↗

W2–W3: Theory and Intra-Class Variation

W2. Theoretical Support for Theorem 2.8

We thank the reviewer for the thorough examination. Theorem 2.8 primarily provides a nonconvex stationarity bound for FedCF's local client optimization. Lemma 2.7 characterizes the optimal representation approximation of local continuous fields relative to class prototypes. However, FedCF's actual teacher comprises local fields, feature and coordinate Translators, the global continuous field, and the global classifier; its gradient discrepancy should account for their joint effects.

Importantly, Appendix E.4 already derives a general descent bound using the full teacher-gradient discrepancy in Eqs. (34)–(47). In the revised manuscript, Eqs. (48)–(49) and Eq. (10) retain this discrepancy directly, rather than replacing it with a bound based solely on the optimal local-field approximation error. Accordingly, following Eq. (47), we have revised the teacher-error term in Eq. (10) as follows:

κ2ϵr2⟶1TE∑τ=0TE−1E∣∣δτ∣∣22\kappa^{2}\epsilon_{r}^{2} \longrightarrow \frac{1}{TE}\sum_{\tau = 0}^{TE - 1}\mathbb{E}\left||\delta_{\tau}\right||_{2}^{2}

Here, δτ\delta_\tau, defined in Eq. (34), measures the full gradient discrepancy between the learned and ideal teachers, accounting for the combined effects of the global fields, feature and coordinate translators, and global classifier. The revised theorem and Appendix E.4 consistently retain this term without introducing additional error-propagation assumptions.

The revised Eq. (10) still supports FedCF's client optimization. Under the stated conditions, more training iterations reduce the initial-gap term in the optimization-error upper bound, while more independently sampled spatial queries reduce gradient-estimation variance, improving the local optimization-error bound. The bound also depends on the gradient discrepancy between actual and ideal teachers, indicating that more accurate global-teacher knowledge can further improve client optimization. Independently, Theorem 2.5, Corollary 2.6, and Lemma 2.7 establish the spatial low-rank property of continuous fields and their no-worse optimal representation approximation than a single class prototype. In summary, the revised Theorem 2.8 still provides a conditional nonconvex stationarity bound for cross-round client optimization under a dynamically updated global teacher.

W3. Intra-Class Spatial Structure and Sample-Level Variation

We thank the reviewer for the detailed comments. FedCF extracts class-level spatial patterns shared across samples through continuous fields while retaining independent feature learning from real inputs on each client.

1. Spatial Structure vs. Sample Variation

Under Assumption 2.2 and Eq. (3), FedCF learns spatially varying class-expected features, not full sample-level feature distributions. Unlike conventional prototypes that compress each class into one vector, FedCF uses shared learnable coordinate factors and low-rank cores to model spatial variation and dependencies among locations, providing richer spatial knowledge. Clients still extract features from their own inputs: Eq. (7) provides feature-alignment supervision, while Eq. (9) retains cross-entropy classification with true labels and prediction distillation. Thus, class-level spatial knowledge sharing complements sample-level discriminative learning.

2. Direct Evaluation on an Independent Test Set

Fig. 3 shows spatial variation in real-sample features, and Fig. 4 demonstrates the reconstruction advantage of continuous fields over class prototypes. On an independent CIFAR-100 test set, we further analyze spatial variation in class-mean features, residuals of individual samples relative to class means, and reconstruction errors of both representations on unseen features.

Table R5: Held-out test-set feature reconstruction and variance decomposition

Feature spaceSpatial-mean varianceSample-residual varianceReal-sample MSE ↓ (Prototype / FedCF)Spatial-mean MSE ↓ (Prototype / FedCF)
Local feature space0.1600.0900.264 / 0.1750.174 / 0.085
Global canonical space0.0320.0100.046 / 0.0270.036 / 0.017

In both local and global canonical spaces, FedCF achieves lower real-sample and spatial-mean reconstruction errors than class prototypes. Both spaces also exhibit nonzero spatial-mean and sample-residual variances, showing that actual features of different images within a class remain distinct. Through continuous low-rank representations, FedCF shares spatial structure that a single class prototype cannot capture while preserving feature learning from real sample inputs, making shared class knowledge and sample-level discrimination complementary.

AuthorsOpenReview ↗

W4: Representation and CAM Fairness

W4. Fairness of Representation Comparisons and CAM Visualization

We thank the reviewer for the suggestion. We add controlled representation comparisons and clarify CAM evaluation to isolate the contribution of continuous low-rank modeling.

1. Advantages of Continuous Low-Rank Modeling

Fig. 6(a) already shows the benefit of spatial structure over a single class prototype. FedCF uses shared learnable continuous coordinate functions and low-rank class cores to model dependencies among spatial locations. Rather than storing features independently at each location, shared continuous functions model them jointly. Thus, observations across locations help learn spatial variation, reduce reliance on independent per-location statistics, and allow flexible queries at different resolutions.

To verify these benefits, we conduct controlled CIFAR-100 experiments within the same FedCF collaboration framework, replacing the knowledge representations at both clients and server while keeping training settings identical. Two discrete variants also match FedCF's total uplink and downlink communication budget.

Table R6: Controlled comparison of knowledge representations

RepresentationDesignAccuracy (%) ↑Comm. (MB/round)
Class prototypeOne mean vector per class54.8514.91
Per-position class meanClass means computed independently at each location56.3264.75
Native learnable spatial gridLearns positional features at the original resolution56.8064.75
Budget-matched spatial gridIncreased grid capacity to match communication57.4371.02
Budget-matched discrete low-rankSame low-rank core, discrete factor tables, and linear interpolation57.9171.03
FedCFShared continuous coordinate functions and low-rank class cores59.3071.07

Spatial means outperform a single class prototype, and learnable grids further improve performance, confirming the roles of spatial information and representation capacity. More importantly, under nearly identical communication budgets, FedCF exceeds the spatial-grid and discrete low-rank baselines by 1.87 and 1.39 percentage points, respectively. Thus, its gains arise not only from additional spatial locations or low-rank decomposition but also from learnable continuous coordinate functions that adaptively model spatial variation. FedCF contributes beyond sharing richer class-level spatial information: it effectively organizes this knowledge through shared learnable continuous functions and low-rank structure, enabling better cross-architecture collaboration.

2. Fairness of CAM Visualization

To address the different scoring scalars in Appendix Eqs. (70) and (71), we unify the scoring of Proto-CAM and Field-CAM. Using the same trained FedCF model and identical test images, both first map per-location features through the same Feature Translator, compute cosine similarity at each location, and then average spatially. The sole difference is the reference: Proto-CAM uses one class prototype at every location, whereas FedCF uses continuous-field features at the corresponding coordinates.

Revised Proto-CAM (Eq. 70):

sproto(x,c)=1∣Ω(x)∣∑p∈Ω(x)cos⁡(Ai(Φi(x)p),μcglob)s_{\mathrm{proto}}(x,c)=\frac{1}{|\Omega(x)|}\sum_{p\in\Omega(x)}\cos\left(A_i(\Phi_i(x)_p),\mu_c^{\mathrm{glob}}\right)

Unchanged Field-CAM (Eq. 71):

sfield(x,c)=1∣Ω(x)∣∑p∈Ω(x)cos⁡(Ai(Φi(x)p),Sc(ψi(p)))s_{\mathrm{field}}(x,c)=\frac{1}{|\Omega(x)|}\sum_{p\in\Omega(x)}\cos\left(A_i(\Phi_i(x)_p),S_c(\psi_i(p))\right)

We have regenerated paired CAMs using unified scoring and updated Figs. 5 and 7 in the revised manuscript. The new visualizations show that Field-CAM's high-response regions are more concentrated on the target object, providing further visual evidence of continuous fields' advantage in representing location-dependent semantic information.

Reviewer_NbKNOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies model-heterogeneous federated learning, where aggregating model parameters across architectures is infeasible. It proposes FedCF, which uses class-conditional low-rank continuous feature fields as a compact knowledge carrier in place of class prototypes or shared proxy models. Client-specific feature and coordinate translators map local fields into a canonical space; the server learns global fields and a classifier, and clients receive feature-level alignment and logit-level distillation. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet with non-IID partitions show clear gains over prototype-based methods and competitive accuracy with proxy-based methods at lower communication cost than the latter.

Critical strengths
  1. The paper identifies a meaningful expressiveness-efficiency trade-off between class prototypes and proxy models. Sharing spatially varying, low-rank feature fields is a coherent response to this bottleneck; the canonical-space alignment and two-level distillation directly serve this central design rather than appearing as unrelated additions.
  2. In Table 1, FedCF improves CIFAR-100 (Pat) accuracy from 59.49% for the strongest prototype baseline to 64.02%, and achieves the best reported accuracy on all three datasets under this partition. Its communication cost is higher than prototype sharing but much lower than proxy-model exchange. The rank-1 ablation in Figure 6(a) and the reconstruction results in Figure 4 support the importance of retaining spatially varying information.
Critical weaknesses
  1. Under the squared-error fitting objective in Eq. (3), each class field primarily estimates a coordinate-conditioned mean feature rather than modeling sample-specific modes. Thus, the improvement may arise mainly from retaining class-wise spatial statistics, not necessarily from continuity or low-rank function modeling per se. A communication-matched spatial-prototype or low-rank discrete-grid baseline would make the claimed contribution substantially more convincing.
  2. Eq. (5) jointly fits global fields and learnable translators using a least-squares objective. In principle, constant class-indistinguishable translated features and global fields can minimize this alignment loss, even if translated features are normalized. The strong empirical performance suggests that such degeneration is not dominating training, but the paper does not explain what prevents it or show evidence of class separation in the canonical space. This matters because mitigating feature collapse is part of the central motivation.
Questions for authors
  1. Can the authors compare FedCF with a spatial-prototype or low-rank discrete-grid baseline under a comparable communication budget and, ideally, the same alignment and distillation pipeline? This would help determine whether the improvement specifically requires continuous-field modeling.
  2. What prevents the global-field/translator optimization in Eq. (5) from converging to class-indistinguishable representations? Are there implicit constraints or class-discriminative gradients acting on the translators, and can the authors provide a class-separation diagnostic in the canonical space?
3 comments
AuthorsOpenReview ↗

Q1: Continuous Fields and Matched Baselines

Q1. Independent Contribution of Continuous Feature Fields and Communication-Matched Comparisons

We thank the reviewer for this important question. We clarify the method, empirical evidence, and theory.

1. Key Methodological Advantages

Eq. (3) learns class-wise expected features at different spatial locations, not full sample-level feature distributions. While class prototypes compress each class into one vector, FedCF preserves spatial variation in class features. Fig. 3 visualizes this structure, and Fig. 4 shows its reconstruction advantage over class prototypes.

Beyond preserving spatial statistics, FedCF models and shares them via learnable continuous low-rank functions, offering:

a. Non-IID robustness: Shared coordinate basis functions link spatial locations, while low rank limits complexity, helping mitigate noise and overfitting from independently estimating each location with limited client data.

b. Spatial expressiveness: Unlike storing features independently at each location or using fixed interpolation, learnable continuous coordinate functions adaptively capture spatial variation with controlled capacity.

c. Heterogeneous knowledge sharing: Continuous functions allow flexible queries across resolutions; with Translators, architectures such as CNNs and ViTs share knowledge in a unified space.

2. Five Baselines and Experimental Validation

To isolate the contribution of continuous function parameterization, we conducted controlled CIFAR-100 experiments. All settings are identical, with only the knowledge representation consistently replaced. We further match the communication budget of FedCF for spatial-grid and discrete low-rank variants.

Table R1: Controlled comparison of knowledge representations

MethodDesignAccuracy (%) ↑Comm. cost (MB/round)
Class prototypeOne mean vector per class54.8514.91
Spatial meanPer-location mean for each class56.3264.75
Native spatial gridTrainable feature grid56.8064.75
Comm.-matched spatial gridIncreased grid capacity to match budget57.4371.02
Comm.-matched discrete low-rankDiscrete coordinate factors, low-rank core, and interpolated queries; matched budget57.9171.03
FedCFLearnable continuous coordinate factors and low-rank core59.3071.07

Spatial means outperform class prototypes by 1.47 points, confirming the value of spatial information. A trainable grid adds 0.48 points, and greater capacity adds another 0.63 points, showing that learnability and capacity also matter. Crucially, FedCF gains 1.87 points over the communication-matched spatial grid and 1.39 points over the communication-matched discrete low-rank method. The latter also controls for low-rank structure, indicating that gains do not stem solely from low-rank decomposition but also from how learnable continuous coordinate functions adaptively organize spatial knowledge.

Corollary 2.6 identifies class prototypes as a special case of continuous fields; Lemma 2.7 establishes that continuous fields have no greater optimal approximation error than class prototypes; Theorem 2.5 guarantees spatial low-rank structure across sampling resolutions. Thus, FedCF contributes beyond retaining class-wise spatial statistics by effectively organizing and sharing spatial knowledge via shared learnable continuous coordinate functions and low-rank structure. These controlled comparisons have been incorporated into Sec. 3.4 and Appendix G.4 of the revised manuscript.

AuthorsOpenReview ↗

Q2: Alignment Collapse and Class Separability

Q2. Class Collapse in Eq. (5) and Class Separability in Canonical Space

We thank the reviewer for raising this important issue and clarify the potential class collapse in Eq. (5).

1. Optimization Mechanism

FedCF's global alignment builds on class-semantic local features. Supervised client backbones learn class-discriminative features; Eq. (3) then fits class-wise expected spatial features, continuously updated during local training, to provide class-aware semantic references. Eq. (5) alternates between two steps: with the Translator fixed, the global field learns mapped features for each class; with the global field fixed, the Translator updates toward class-conditional global targets. If the fixed class references are discriminative, mapping all classes to one constant cannot match all targets, yielding nonzero alignment error. Thus, the Translator receives class-dependent gradients from Eq. (5)'s class-conditional targets, without an extra classification loss.

2. Empirical Evidence

To test for class collapse, we added the following CIFAR-100 diagnostics:

Table R2: Canonical-space class separability and semantic preservation

MetricFully collapsed (theoretical)FedCF
Global-field inter-class cosine similarity ↓10.62
Global-field class-center distance ↑00.86
Effective rank of centered class centers ↑029.6
Cross-architecture semantic gap ↑00.25
Classification accuracy on real test features ↑≈1%42.0%

The inter-class similarity, class-center distance, and effective rank show that classes do not converge to one constant but retain multidimensional differences. The cross-architecture semantic gap (0.25) indicates that mapped real test features from different architectures exhibit higher average cosine similarity for same-class pairs than for different-class pairs. Crucially, we pass independent test images through client backbones and the feature Translator directly into the frozen global classifier. Without using ground-truth labels as input or querying the global field by class, this yields 42.0% accuracy, far above the ≈1% chance level for 100 classes. Thus, class-discriminative information is preserved not only in class-conditioned global-field representations but also in mapped features of real images.

Fig. 6(b) already shows that alternating optimization achieves higher accuracy and lower alignment error than joint optimization. Under identical settings, we further measured the variance and magnitude retention ratios of both schemes, quantifying how well the global field preserves the variation and magnitude of locally mapped features (values closer to 1 indicate better preservation). Alternating optimization achieves 0.849 and 0.878, respectively, outperforming joint optimization (0.625 and 0.554). Together with the above class-separability diagnostics, these results further demonstrate that FedCF effectively mitigates representation degeneration during training through supervised class-semantic references and alternating optimization.

Dear Reviewer NbKN,

Thank you again for your careful review and valuable feedback.

We have addressed your comments in our rebuttal, added experiments and analyses, and uploaded a revised manuscript.

As the discussion period draws to a close, we would appreciate knowing whether our responses and revisions have resolved your main concerns. If any questions remain, we would be happy to provide further clarification or discussion.

Thank you again for your time and support!

Best regards,

The Authors

Comments on the paper

1 comment

We sincerely thank all reviewers for their careful evaluation and valuable feedback, which has been essential to improving our work. We have revised and supplemented the manuscript in response to the reviewers' comments. The revised PDF has been uploaded, with all changes highlighted in blue.

To avoid confusion regarding numbering, all references to figures, tables, equations, and appendices in our responses follow the latest revised manuscript. Additional experimental tables presented in the rebuttal use separate numbering (Tables R1–R13) to distinguish them from those in the manuscript. Some of these experimental results have also been incorporated into the revised manuscript and may therefore appear under different table numbers.

We again thank all reviewers for their thorough evaluation and constructive suggestions. We hope these revisions adequately address the concerns raised and make the manuscript clearer, more rigorous, and more complete.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.