acceptodds
Under review as a conference paper at ICLR 2027

RGM-SL: Receiver-Aware Gradient Matching for Socialized Generalist Cultivation

Abstract

Socialized learning brings together domain experts to cultivate across-domain generalist and return its integrated knowledge to the expert community. Under pathological label partitions, this process must transfer complementary local knowledge as the generalist's learning needs evolve. We introduce RGM-SL, a receiver-aware gradient matching approach to socialized generalist cultivation. After an initial bootstrap stage, each expert uses the current generalist as the receiving model and optimizes a compact synthetic dataset to align real- and synthetic-data gradients under its training objective. This makes the transmitted data responsive to both local expert knowledge and the generalist's current state. The framework coordinates model-level receiver states, sample-level synthetic data, and label-level supervision. The server aggregates expert uploads and trains the generalist with hard labels, class-masked soft labels, and expert anchoring. The resulting shared representation supports lightweight feedback heads that extend the experts' cross-domain capabilities. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet show that the complete receiver-aware refinement procedure improves overall generalist accuracy over random real-sample selection at matched images-per-class and round budgets on all three datasets. Generalist feedback further improves community overall accuracy and enables every expert to recognize classes outside its original domain.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
socialized learning, communication-efficient learning, multi-granularity interaction, dataset distillation, gradient matching, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CwHwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RGM-SL studies communication-constrained socialized learning under disjoint label partitions. After expert-based bootstrap, each expert refines a small synthetic image packet by matching real- and synthetic-data gradients on the current generalist, with teacher classification and feature regularization. The server trains with hard labels, masked soft targets, and reference-expert anchoring. A frozen generalist subsequently supports local feedback heads and a router. Evaluation uses ConvNet3 on CIFAR-10, CIFAR-100, and Tiny-ImageNet, with budgeted baselines and a matched-IPC Random Real comparison.

Critical strengths
  1. The receiver-conditioned formulation is well motivated: Eqs. (9)–(13) adapt transmitted examples to the current generalist rather than optimizing only for a separate expert. The formulation clearly identifies the optimized pixels and fixed receiving model.

  2. The complete refinement pipeline improves Overall accuracy over Random Real on all three datasets at the same IPC and round count: 47.61 to 49.02, 35.44 to 37.62, and 20.66 to 21.06 (Table 4). This provides evidence that refining packet content can be useful without adding image slots.

  3. The evaluation separates generalist performance from routed community performance, reports local versus non-local class accuracy, and uses validation-selected checkpoints with held-out test sets. These distinctions make the intended tradeoffs easier to inspect.

Critical weaknesses
  1. Inconsistent evaluation numbers. Under the stated balanced CIFAR-10 test set and 5-by-2 partition, Overall must equal 0.2 times Expert plus 0.8 times Non-expert. Table 1 gives FedAvg as 55.90/45.96/48.54, but these subset values imply 47.95 Overall. FedDM similarly implies 47.77, versus the reported 48.46. These gaps exceed rounding and leave the headline baseline comparisons insufficiently verified.

  2. The central receiver-aware mechanism is not isolated. Table 4 compares the complete pixel-refinement pipeline with no pixel optimization, while Eq. (12) also includes teacher classification and feature-statistic regularization. It does not establish whether updating the receiver state is responsible for the gain. A fixed-receiver comparison and a zero receiver-matching-weight control with the auxiliary terms retained would directly test the main contribution.

  3. Small gains are supported by only one seed. Section 5.1 and Appendix A explicitly report seed 42 only. The Overall margins over FedAvg are 0.48 points on CIFAR-10 and 0.75 on Tiny-ImageNet; the Tiny-ImageNet gain over Random Real is 0.40 points. Without repeated runs and uncertainty estimates, the robustness of these improvements is unresolved.

  4. Communication efficiency is not quantitatively demonstrated. Eqs. (19)–(23) define accounting, but the paper does not tabulate actual uplink/downlink totals, transport precision, or the FedAvg/FedProx round counts admitted by the budget. One-shot methods instead match image counts, which does not establish equal byte budgets. Without these values or accuracy-versus-bytes results, the claimed resource tradeoff cannot be assessed.

  5. The added value of feedback remains unclear. Table 2 shows mean Expert accuracy falling from 95.58 to 70.94, 56.45 to 44.95, and 43.93 to 32.22. Community Overall accuracy after feedback is also below using the generalist alone, and Table 3 reports substantially stronger MASC results. Nonzero unseen-class accuracy is partly supplied by routing to the generalist. A generalist-only and an unchanged-expert-plus-generalist routing control are needed to establish what the trained feedback heads contribute.

Questions for authors

Clarifications that could change the assessment:

  1. Can you reconcile the inconsistent Table 1 subset and Overall accuracies, specifying test counts and confirming that all three metrics come from the same checkpoint and predictions?

  2. With the other losses and packet initialization fixed, how much do the current receiver state and the receiver-gradient term each contribute? Do the small gains persist across repeated seeds?

  3. What are the actual byte budgets and eligible baseline round counts? At comparable end-to-end cost, what benefit do the feedback heads provide over using the generalist alone or routing between the unchanged expert and generalist?

Reviewer_ZdbTOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

RGM-SL is a socialized learning method that aggregates the knowledge of multiple experts—each trained only on different class domains—to create a single cross-domain generalist. Synthetic images are adjusted so that the gradients induced by the experts’ real data on the current generalist are as similar as possible to those induced by synthetic data, and these images are then sent to the server along with hard labels, class-masked soft labels, and expert IDs. The server aggregates the synthetic packets from all experts and updates the generalist using mixed hard–soft supervision and anchoring. Finally, it transmits the generalist’s updated representation back to each expert, enabling them to recognize classes they were not originally trained on. In experiments, the method demonstrated higher overall accuracy on CIFAR-10, CIFAR-100, and Tiny-ImageNet compared to the Random Real method under the same IPC and communication round conditions.

Critical strengths
  1. Rather than simply compressing the expert’s local data, the synthetic data is optimized based on the gradient updates that the generalist actually requires at any given moment.
  2. By transmitting synthetic images instead of curated data, the data can be tailored to enable the most effective updates, which may also help address privacy concerns to some extent.
  3. Since the final Generalist’s representation is fed back to the Expert to train the lightweight head, even Experts that previously could not recognize classes outside their original domain gain cross-domain recognition capabilities.
Critical weaknesses
  1. The performance improvement is marginal relative to the high additional computational cost. While the overall accuracy is the highest, the non-expert accuracy is lower than that of FedAvg on all three datasets, and the overall gain is also small (Table 1). Furthermore, the gain is similarly small in experiments comparing it to random real data (Table 4). Since every expert must perform receiver-aware gradient matching and iteratively optimize synthetic images during each communication round, the performance improvement gained is too small relative to the additional computational overhead incurred for gradient calculations and pixel-level optimization.
  2. While the paper provides a detailed comparison of communication budgets, it does not quantitatively compare computational overheads, such as FLOPs and energy consumption, associated with generating synthetic packets. Therefore, it remains unclear whether the “communication-efficient” advantage actually translates into overall cost savings at the system level.
  3. The scalability validation is insufficient, as it relies solely on CIFAR-10, CIFAR-100, and Tiny-ImageNet. Experiments with larger datasets, such as ImageNet (or subsets like ImageNet-200), and stronger backbones are needed.
  4. The concept of determining which data to transmit based on the needs of the current global model, as seen in “Take What I Need” (Li et al., 2025), already exists. Although RGM-SL differs in that it is implemented using gradient matching, its high-level novelty is limited.

Li, H., Zhou, Y., Wang, Y., Gu, X., Li, B., & Wang, W. (2025, June). Take What I Need: Active Data Distillation for Federated Learning. In 2025 IEEE International Conference on Multimedia and Expo (ICME) (pp. 1-6). IEEE.

Questions for authors

Please refer to the weaknesses.

Reviewer_SctHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies socialized learning: several domain experts with disjoint classes cooperate to cultivate a generalist covering all classes, whose knowledge is then fed back to the experts. RGM-SL initializes the generalist from a reference expert; in each round, every expert refines a small packet initialized from local real images by gradient matching on the current generalist under the generalist's mixed hard–soft loss and uploads it, and the server trains the generalist on the packets ("receiver awareness"); finally, each expert trains a local head on the frozen generalist features. Experiments use ConvNet3 on CIFAR-10, CIFAR-100 and Tiny-ImageNet (single seed), where RGM-SL attains the highest Overall and Expert accuracy.

Critical strengths

The idea is natural. Since the generalist keeps changing during training, tailoring the uploaded synthetic data to the receiver's current training objective, rather than condensing once at a fixed initialization, is an intuitive and reasonable starting point.

The method and the evaluation protocol are described fairly completely. Algorithm 1, the communication accounting formulas (Equations (19)–(21), (23)), the hyperparameters (Table 5) and the evaluation protocol (Appendix C) are all clearly stated, including that Overall is weighted by samples rather than being a simple average of the two columns.

The compared methods cover three categories — parameter exchange (FedAvg, FedProx), one-shot condensation (FedD3, FedSD2C) and multi-round synthetic data (FedDM, FedAF, FedUSD) — and a Random Real control with the same IPC and number of rounds is included. The authors also honestly report that MASC is clearly better in the feedback stage.

Critical weaknesses

Location: Table 1 (lines 341–355), Section 4.2 (line 155), Equation (17) (lines 256–262), metric definitions in Section 5.1 (lines 328–329), Appendix C (lines 696–697) Concern: This is my main concern: the advantage of RGM-SL in Table 1 comes mainly from the classes of the reference expert, and these classes receive special treatment in the method. The Expert column is computed only on the reference expert's classes (lines 328, 696), the generalist is initialized from precisely this reference expert (line 155), and training includes an anchoring distillation term that acts only on (Equation (17)). The paper does not say whether the baselines are also initialized from the same reference expert or use an anchoring term; if not, the comparison in the Expert column is not fair. On the remaining domains, which make up 75%–80% of the test set (Non-expert column), RGM-SL is worse than FedAvg on all three datasets (42.2 vs. 45.96, 33.91 vs. 35.24, 17.84 vs. 20.88), and ranks only fourth on CIFAR-10 and CIFAR-100. The lead in Overall (+0.48, +2.06 and +0.75 over FedAvg) is essentially bought with the large lead on the reference classes. For a generalist meant to cover all domains, the average or the worst-case accuracy over domains would be a more appropriate metric. I suggest giving the baselines the same reference-expert initialization and anchoring, or reporting RGM-SL without initialization and anchoring, and providing per-domain accuracies to show whether the conclusions depend on which expert is chosen as the reference.

Location: Section 4.3 (lines 199–234), pixel constraint in Appendix A (lines 682–686), Table 4 and Section 5.4 (lines 445–473), Section 2 (lines 084–098) Concern: The core claim in the title, "receiver awareness," is not validated on its own, and the uploaded data are essentially the original images. In every round the synthetic images are initialized from local real images and projected onto an neighborhood of radius (about 13/255) (line 683); in other words, what is uploaded are real images with small perturbations, 150 per class over 15 rounds (plus the bootstrap packet). The authors state that this projection is not a privacy mechanism (line 686), yet line 308 emphasizes that the experts' data always remain local; the two statements are not quite consistent. In this setting, the most natural control is to upload the real images directly, and in Table 4 the improvement of RGM-SL over Random Real is only +1.41, +2.18 and +0.40 points. More importantly, this control compares the entire refinement pipeline (the authors also note that it does not separate the individual loss terms, lines 462–463), so it cannot show that "matching gradients on the current generalist" is better than "matching on a fixed model." At minimum the following are needed: the same gradient matching on a fixed model (the expert itself, the initial generalist , or a randomly initialized network); refining once and reusing the same packet in every round; and setting , keeping only the teacher cross-entropy and feature terms. To my understanding, FedDM also constructs synthetic data around the received global model in every round, which is close to the receiver-aware idea of this paper; I suggest explaining the difference between the two in the main text. In addition, since the uploaded data are essentially the original images, the privacy difference should be discussed when comparing against FedAvg and FedProx, which share no data.

Location: Tables 1 and 4 Concern: The key differences are all small: in Overall, RGM-SL leads the strongest baseline by only 0.48 / 2.06 / 0.75 points and Random Real by only 1.41 / 2.18 / 0.40 points. Moreover, the numbers in Table 1 are internally inconsistent. Since the test sets are class-balanced, Overall should be exactly the class-count-weighted average of Expert and Non-expert (0.2/0.8 for CIFAR-10 and Tiny-ImageNet, 0.25/0.75 for CIFAR-100). All rows for RGM-SL and Random Real satisfy this, but five baseline rows do not: FedAvg on CIFAR-10 (47.95 computed from the two columns vs. 48.54 in the table), FedUSD (42.20 vs. 42.85) and FedDM (47.77 vs. 48.46), as well as FedAvg (20.42 vs. 20.31) and FedProx (16.63 vs. 16.78) on Tiny-ImageNet. On CIFAR-100, all three columns of FedD3 are 35.36, which should also be checked. Please clarify whether the Overall of these baselines comes from a different checkpoint than the other two columns.

Location: Abstract (lines 019–020), Section 1 (lines 051–052, 064), Section 4.5 (lines 270–285), Section 5.1 (lines 328–329), Tables 1–3, Appendix C (lines 696–700) Concern: Two writing issues affect the understanding of the contributions and results. (1) The abstract, the introduction and the list of contributions all present the "multi-granularity" coordination at the model, sample and label levels as one of the contributions (lines 019–020, 051–052, 064), but Section 4.5, titled Multi-Granularity Interaction (lines 270–285), contains only the communication accounting formulas, and no concrete mechanism corresponding to "multi-granularity" can be found in the paper, so this contribution lacks substantive support. (2) The Expert column in Table 1 refers to the reference expert's classes (lines 328–329, 696–697), whereas the Expert column in Tables 2 and 3 refers to each expert's own classes (lines 699–700): the column name is the same but the meaning differs. The Expert column is exactly what my first concern hinges on, and this mixed usage can easily lead readers to misread the results; I suggest distinguishing the two in the table headers or captions.

Questions for authors

Are the baselines also initialized from the reference expert, and do they use the anchoring term? What are the three columns for RGM-SL without reference-expert initialization and anchoring? What are the average and worst-case accuracies over domains? How do the variants with gradient matching on a fixed model, with the same packet reused in every round, and with compare to the full method? What is the essential difference from FedDM, which constructs synthetic data based on the global model in every round? In Table 1, why does Overall not match the Expert and Non-expert columns for FedAvg, FedUSD and FedDM on CIFAR-10 and for FedAvg and FedProx on Tiny-ImageNet?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.