acceptodds
Under review as a conference paper at ICLR 2027

Heterogeneous Socialized Learning: Enabling Individual Growth Across Diverse Model Architectures

Abstract

Collaborative learning among heterogeneous agents offers a promising approach to overcoming the limited knowledge of individual agents. However, existing heterogeneous federated learning methods primarily focus on improving global or local task performance, rather than enabling each agent to acquire complementary knowledge beyond its original expertise. Although socialized learning emphasizes individual knowledge acquisition and preservation, its reliance on shared model architectures limits its applicability to heterogeneous agents. To address these challenges, we formulate Heterogeneous Socialized Learning, a new paradigm that enables agents with distinct model architectures to acquire complementary knowledge from their peers while preserving their original expertise. We further propose Heterogeneous Socialized Collaboration (HSC), an architecture-agnostic framework that facilitates cross-model knowledge exchange through compact knowledge packets constructed via dataset distillation. By integrating peer knowledge distillation with feature retention, HSC enables each agent to acquire new knowledge while retaining its existing expertise and model architecture. Extensive experiments on CIFAR-100 and Tiny ImageNet demonstrate that HSC consistently improves new knowledge acquisition and overall competence across diverse model architectures and agent populations while retaining substantial original expertise.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
socialized learning, model heterogeneity, multi-agent collaboration, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cQywOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates Heterogeneous Socialized Learning (HSL), where agents with different model architectures are expected to acquire complementary knowledge from peers while retaining their original expertise and architecture. To instantiate this paradigm, the authors propose Heterogeneous Socialized Collaboration (HSC). Each agent first condenses its private expertise into a knowledge packet containing synthetic samples, labels, and expert logits. A receiver then learns complementary classes using hard-label supervision and peer knowledge distillation while applying feature retention against a frozen copy of its original encoder. Experiments on CIFAR-100 and Tiny ImageNet with 5, 10, and 20 agents show improved New and Global accuracy over the selected baselines.

Critical strengths
  1. Instead of optimizing only a global model or local-task accuracy, HSL explicitly focuses on whether every heterogeneous agent can acquire peer expertise while retaining its own architecture. This is an interesting perspective on heterogeneous multi-agent collaboration.

  2. Exchanging synthetic samples and class-indexed logits avoids direct alignment of heterogeneous parameters or feature dimensions, making the framework conceptually applicable to receivers with different backbones.

Critical weaknesses
  1. The paper defines HSL as enabling each agent to acquire complementary knowledge while preserving its original expertise. However, the reported Expert accuracy drops substantially after collaboration. For example, on CIFAR-100, HSC decreases from 74.89 to 61.05 with 5 agents, from 80.48 to 57.75 with 10 agents, and from 85.49 to 57.14 with 20 agents. Similar degradation is observed on Tiny ImageNet. This level of forgetting is difficult to reconcile with the repeated claim that HSC “retains substantial original expertise.” Since expertise preservation is one of the two central objectives of the paper, this issue directly weakens the main conclusion.

  2. HSC mainly combines several established techniques, including dataset condensation, hard-label learning, knowledge distillation, and feature retention. The condensation stage explicitly follows DSDM, while the internalization stage largely relies on standard KD and cosine feature regularization. Although the HSL formulation provides a different perspective, the technical contribution of HSC itself is not sufficiently distinguished from existing heterogeneous FL, dataset-distillation, and knowledge-transfer methods.

  3. The experiments mainly compare against DeSA, FedRE, Heuristic, and FAST, while MASC cannot actually operate in the heterogeneous setting and only contributes pre-collaboration Expert accuracy. At the same time, the paper cites several highly relevant heterogeneous or one-shot FL methods, including FedMD-style distillation, synthetic-anchor methods, generative heterogeneous FL, and recent feature-distillation approaches, but does not evaluate them. Moreover, the near-zero New accuracy of DeSA and FedRE may partly reflect a mismatch between their original objectives and the paper’s class-disjoint acquisition setting rather than a decisive advantage of HSC. Stronger task-matched baselines are necessary for a fair comparison.

  4. Evaluation is limited to CIFAR-100 and Tiny ImageNet with five relatively conventional CNN architectures, and the class space is evenly partitioned into strictly disjoint subsets across agents. This setup excludes many realistic forms of heterogeneity, such as partially overlapping labels, unbalanced class coverage, unequal local dataset sizes, domain shifts, partial participation, and heterogeneous device capacities. The paper motivates HSL using real-world applications such as autonomous systems, healthcare, and IoT, but the empirical study does not reproduce these conditions. As a result, the practical generalizability of the framework remains unclear.

  5. HSC requires every agent to perform dataset condensation and then exchange synthetic samples and expert logits with peers. However, the paper does not report packet size, communication volume, condensation time, receiver-side training cost, memory overhead, or how these quantities scale with the number of agents, classes, and synthetic samples per class. This omission is particularly problematic because the paper explicitly motivates the method with resource-constrained agents. In addition, distilled synthetic samples may still reveal private information, yet no privacy analysis or leakage evaluation is provided. Thus, the claimed practical suitability of HSC is insufficiently substantiated.

  6. The ablation only removes KD or FR, while many other important components remain unexamined, including packet construction, expert logits, classifier prototype initialization, the self-packet, the mixing coefficient \rho, the number of synthetic samples K, \lambda_{KD}, \lambda_{FR}, temperature \tau, and the condensation objective. More importantly, removing FR changes Global accuracy by only −0.03-0.03 percentage points, even though feature retention is presented as the key mechanism for preserving existing expertise. This weak empirical effect raises doubts about whether the proposed retention component plays the central role claimed by the paper. A much more complete component-wise and sensitivity analysis is required.

Questions for authors

How do the authors justify the claim of “expertise preservation” given the substantial drop in Expert accuracy after collaboration, especially when the number of agents increases?

What is the key methodological novelty of HSC beyond combining dataset condensation, hard-label learning, knowledge distillation, and feature retention?

How sensitive is HSC to the construction of the knowledge packet?

Reviewer_cx9dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper present how agents with different model architectures can collaboratively learn new knowledge and retaining their original capabilities. The authors propose HSL with HSC method. Each agent first condenses its local knowledge into a small amount of synthetic data, class labels, and predictions, and then uses KD and FR to enable other agents to learn this knowledge without altering their own model architectures. The authors conducted experiments on CIFAR-100 and Tiny ImageNet using five model architectures, testing scenarios with 5, 10, and 20 agents. HSC achieved the highest New and Global accuracy across all experimental settings. Ablation experiments also demonstrated that KD facilitates the acquisition of new knowledge, while FR helps minimize the loss of existing knowledge.

Critical strengths
  1. The research question addressed in the paper is meaningful. Unlike traditional federated learning, this study focuses on whether each agent can acquire new knowledge, rather than merely improving the performance of the global model.
  2. The methodological design is well. By exchanging synthetic data and prediction results, models with different architectures can directly learn from one another. This approach requires neither parameter sharing nor modifications to the original model architecture.
  3. The experimental result are promising. HSC demonstrates significant improvement across different datasets, model architectures, and numbers of agents. In addition, it maintains strong new knowledge learning capabilities even as the number of agents increases.
Critical weaknesses
  1. The prior knowledge is retained remains unclear. In the 20-agent experiment on CIFAR-100, HSC’s Expert accuracy was 57.14%, lower than the Heuristic’s 64.87%. Although the Global accuracy was higher, this was primarily due to an improvement in New accuracy. The paper does not directly report changes in accuracy for the same agent before and after collaboration. Therefore, it is difficult to determine how much prior knowledge was actually forgotten.
  2. The method’s innovativeness requires further clarification. HSC utilizes multiple existing techniques. In ablation experiments, remove KD resulted in a decrease of only 0.67 percentage points in Global accuracy, and remove FR caused no change. However, the performance improvement in the main experiment was significantly greater. The authors don't explain which component is responsible for the primary improvement.
  3. Some of the comparative experiments are not sufficiently convincing. The original research objectives of DeSA and FedRE differ with this paper. Under the current setup, the new accuracy of these methods is zero. The authors do not adequately explain the reasons for this. There is also a lack of simple control methods using the same synthetic data and training settings.
Questions for authors
  1. Could you provide the expert accuracy for each agent both before and after collaboration? To what extent does the forgetting of previously acquired knowledge occur? How do the expert accuracy and new accuracy change when the weights for FR are adjusted?
  2. What would the results be if the same synthetic data and training configuration were used and with only cross-entropy then what extent does the initialization method described in Equation (6) impact performance?
  3. Why do DeSA and FedRE struggle in some experiments to learn new classes? Were these methods specifically hyperparameter-tuned for the tasks at hand? Furthermore, did all the methods operate under the same training budget?
Reviewer_BcDUOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a new paradigm named Heterogeneous Socialized Learning (HSL). This paradigm considers a situation in which agents with different model architectures and complementary class expertise learn from one another. To address this task, the authors propose a method named Heterogeneous Socialized Collaboration (HSC). HSC encapsulates each agent’s local expertise into transferable knowledge packets. Then, HSC lets each receiver internalize peer knowledge on its own backbone through three mechanisms. Experimental results show that HSC achieves promising performance.

Critical strengths
  1. The motivation of this paper is reasonable. The idea that moving from a shared generalist to architecture-agnostic individual growth is convincing.

  2. The proposed HSC method is well designed. The proposed modules are technically reasonable and achieve the motivation.

  3. Extensive experimental results show the HSC method achieves promising performance.

Critical weaknesses
  1. Limited novelty in modules. The proposed method mainly consists of dataset condensation based on synthetic samples, logit-based distillation, and feature retention. However, these technologies have been widely used in the existing literature.

  2. Insufficient experiments. The experiments are conducted on small and simple datasets, i.e., CIFAR-100 and Tiny ImageNet. Experiments on larger and more complex tasks should be considered.

  3. Insufficient comparison. The paper should compare with more heterogeneous federated learning methods.

Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.