- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper formulates Heterogeneous Socialized Learning (HSL), where agents with different model architectures are expected to acquire complementary knowledge from peers while retaining their original expertise and architecture. To instantiate this paradigm, the authors propose Heterogeneous Socialized Collaboration (HSC). Each agent first condenses its private expertise into a knowledge packet containing synthetic samples, labels, and expert logits. A receiver then learns complementary classes using hard-label supervision and peer knowledge distillation while applying feature retention against a frozen copy of its original encoder. Experiments on CIFAR-100 and Tiny ImageNet with 5, 10, and 20 agents show improved New and Global accuracy over the selected baselines.
Critical strengths
-
Instead of optimizing only a global model or local-task accuracy, HSL explicitly focuses on whether every heterogeneous agent can acquire peer expertise while retaining its own architecture. This is an interesting perspective on heterogeneous multi-agent collaboration.
-
Exchanging synthetic samples and class-indexed logits avoids direct alignment of heterogeneous parameters or feature dimensions, making the framework conceptually applicable to receivers with different backbones.
Critical weaknesses
-
The paper defines HSL as enabling each agent to acquire complementary knowledge while preserving its original expertise. However, the reported Expert accuracy drops substantially after collaboration. For example, on CIFAR-100, HSC decreases from 74.89 to 61.05 with 5 agents, from 80.48 to 57.75 with 10 agents, and from 85.49 to 57.14 with 20 agents. Similar degradation is observed on Tiny ImageNet. This level of forgetting is difficult to reconcile with the repeated claim that HSC “retains substantial original expertise.” Since expertise preservation is one of the two central objectives of the paper, this issue directly weakens the main conclusion.
-
HSC mainly combines several established techniques, including dataset condensation, hard-label learning, knowledge distillation, and feature retention. The condensation stage explicitly follows DSDM, while the internalization stage largely relies on standard KD and cosine feature regularization. Although the HSL formulation provides a different perspective, the technical contribution of HSC itself is not sufficiently distinguished from existing heterogeneous FL, dataset-distillation, and knowledge-transfer methods.
-
The experiments mainly compare against DeSA, FedRE, Heuristic, and FAST, while MASC cannot actually operate in the heterogeneous setting and only contributes pre-collaboration Expert accuracy. At the same time, the paper cites several highly relevant heterogeneous or one-shot FL methods, including FedMD-style distillation, synthetic-anchor methods, generative heterogeneous FL, and recent feature-distillation approaches, but does not evaluate them. Moreover, the near-zero New accuracy of DeSA and FedRE may partly reflect a mismatch between their original objectives and the paper’s class-disjoint acquisition setting rather than a decisive advantage of HSC. Stronger task-matched baselines are necessary for a fair comparison.
-
Evaluation is limited to CIFAR-100 and Tiny ImageNet with five relatively conventional CNN architectures, and the class space is evenly partitioned into strictly disjoint subsets across agents. This setup excludes many realistic forms of heterogeneity, such as partially overlapping labels, unbalanced class coverage, unequal local dataset sizes, domain shifts, partial participation, and heterogeneous device capacities. The paper motivates HSL using real-world applications such as autonomous systems, healthcare, and IoT, but the empirical study does not reproduce these conditions. As a result, the practical generalizability of the framework remains unclear.
-
HSC requires every agent to perform dataset condensation and then exchange synthetic samples and expert logits with peers. However, the paper does not report packet size, communication volume, condensation time, receiver-side training cost, memory overhead, or how these quantities scale with the number of agents, classes, and synthetic samples per class. This omission is particularly problematic because the paper explicitly motivates the method with resource-constrained agents. In addition, distilled synthetic samples may still reveal private information, yet no privacy analysis or leakage evaluation is provided. Thus, the claimed practical suitability of HSC is insufficiently substantiated.
-
The ablation only removes KD or FR, while many other important components remain unexamined, including packet construction, expert logits, classifier prototype initialization, the self-packet, the mixing coefficient \rho, the number of synthetic samples K, \lambda_{KD}, \lambda_{FR}, temperature \tau, and the condensation objective. More importantly, removing FR changes Global accuracy by only percentage points, even though feature retention is presented as the key mechanism for preserving existing expertise. This weak empirical effect raises doubts about whether the proposed retention component plays the central role claimed by the paper. A much more complete component-wise and sensitivity analysis is required.
Questions for authors
How do the authors justify the claim of “expertise preservation” given the substantial drop in Expert accuracy after collaboration, especially when the number of agents increases?
What is the key methodological novelty of HSC beyond combining dataset condensation, hard-label learning, knowledge distillation, and feature retention?
How sensitive is HSC to the construction of the knowledge packet?