acceptodds
Under review as a conference paper at ICLR 2027

FedOPD: Private On-Policy Distillation of Heterogeneous Client Knowledge into Foundation Models

Abstract

Transferring decentralized domain expertise to a centralized foundation model is critical for federated foundation models (FedFMs). However, existing methods either expose model states via parameter sharing or risk training data leakage through client-side content generation and outcome-reliant feedback. We propose **FedOPD**, an on-policy distillation framework that renders clients entirely *generation-free* and *outcome-free*. In FedOPD, the server dispatches its own reasoning rollouts to domain clients, whose frozen experts passively evaluate the received tokens via teacher forcing to return dense, token-level log-probabilities. This discrepancy directly optimizes the sequence-level reverse KL divergence with an variance bound, bypassing coarse verifiers and memorization leakage. To facilitate practical deployment, we develop **SPBO** to prevent catastrophic forgetting across sequential domains and **TTF** to prune feedback communication by up to 80%. Extensive experiments on mathematical and sequential multi-domain benchmarks show that FedOPD outperforms prior federated baselines and centralized verifier-guided GRPO across 3B and 7B students, while using only 1% of FedDF's communication bandwidth.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Federated Learning, Reinforcement Learning, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_N5ZGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes FedOPD (Federated On-Policy Distillation). The setting is the following: a central server holds a (larger) student LLM while clients run (smaller) LLM. Clients hold data in specialized domains. The main idea is that clients evaluate the server’s own generated responses rather than generating output themselves or rather than directly disclosing model parameters. The underlying assumption is that the server already posses domain-specific (unlabeled) data, required to perform the distillation. FedOSD works in two stages: first local experts (clients’ LLM) are trained on private data, then frozen; second, the local experts evaluate the server-generated trajectories and return token-level log-probabilities, and the server distills knowledge from this feedback.

Critical strengths
  • Making clients passive entities that evaluate the outputs generated by the server-held LLM is an interesting idea, and the proposed distillation protocol may have merit.
Critical weaknesses
  • The paper makes an implicit assumption: the server already possesses domain-relevant, unlabeled prompts. This assumption is not clearly stated or clearly discussed in the paper. So the server’s distillation prompts are taken from the same underlying public dataset used to train clients’ expert. This means that distribution mismatch between public distillation prompts and private training data is limited by construction.
  • Implementation detail of considered baselines are scarce and the paper does not seem to provide reproducible code.
Questions for authors

Q1. How does the server obtain representative prompts without accessing the private client data?

Q2. How does the method perform when the server does not know the clients’ training inputs and is forced to prompt from a different distribution?

Q3. How many public prompts are needed to achieve the reported results, and what is their ratio relative to the clients' private training data? How does FedOPD perform when the server has access to a limited number of prompts or when prompt availability is highly unbalanced across domains?

3 comments
AuthorsOpenReview ↗

Addressing the server-prompt assumption: privacy-filtered, mismatched, and smaller prompt pools

We thank the reviewer for the precise questions. All new results below will be added to either the main text or the appendix of the final version.

W1 & Q1. The prompt assumption and how to obtain prompts privately.

This assumption is fairly standard: federated distillation with an unlabeled transfer set (FedMD, Li & Wang, 2019; FedDF, Lin et al., 2020) and FedGRPO (Zhu et al., 2026) make the same one. It is natural when the value lies in answers and expertise, not in questions, e.g., public question banks whose solutions and verifiers are proprietary, or domain queries a provider already receives. The submission already states it (Sec. 3.1, line 146; App. B.1, line 724). We will make it an explicit assumption in Sec. 3.1.

In particular, we further examine the case where the problem prompts themselves are treated as private and must be protected. Representative prompts can then be obtained in two ways. First, clients can release problem text after privacy filtering, as shown in Table R1. Second, the server can use a pool independent of client data (see our response to Q2–Q3). For the first route, we either remove the problems that contain personally identifiable information (PII) or mask only the PII spans (to be added to Appendix C):

Table R1: FedOPD with privacy-filtered server prompts (fine-grained math, 3B).

Server promptsPII releasedGSM-HMATH500Int-ALInt-GEInt-NTAVG
Originalyes47.3162.2081.5551.9863.7052.51
Remove PII problemsno46.2561.2081.4751.5760.9352.39
Mask PII spansmasked44.9661.0080.8850.9460.9351.69

Removing all PII-bearing problems costs only 0.12 points. Masking costs 0.82, because placeholders make some problems ambiguous (e.g., "[NAME] has 3 more than [NAME]").

References

  • Li, D. and Wang, J. FedMD: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019.
  • Lin, T., Kong, L., Stich, S. U., and Jaggi, M. Ensemble distillation for robust model fusion in federated learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
  • Zhu, G., Gu, H., Fan, L., Yang, Q., and Han, Y. FedGRPO: Privately optimizing foundation models with group-relative rewards from domain clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, 2026.

Q2. Mismatched prompt distribution. We agree that matched prompts limit mismatch by construction, so we test pools of 6,000 prompts drawn independently of the client data (to be added to Appendix C). Domain proportions are sampled from a Dirichlet distribution Dir(β)\mathrm{Dir}(\beta). A smaller β\beta means stronger imbalance; at β=0.1\beta=0.1, about 90.9% of the pool consists of unrelated everyday prompts.

Table R2: FedOPD with server prompt pools that are mismatched to the client data (fine-grained math, 3B).

Server poolGSM-HMATH500Int-ALInt-GEInt-NTAVG
Matched (Original)47.3162.2081.5551.9863.7052.51
Dir β=10\beta=1046.3263.2082.1450.5262.4152.12
Dir β=1\beta=145.4162.4082.4849.9061.3051.74
Dir β=0.1\beta=0.142.1554.2076.7548.2352.4147.59
Student base39.0453.4076.6643.4254.0746.81

Under moderate mismatch (β≥1\beta\ge1), FedOPD stays above every federated baseline that uses matched prompts. Under extreme mismatch, the gain shrinks to +0.78 over the base, because the server rarely asks about the clients' domains. We report this as a limitation.

Q3. Prompt budget and imbalance. The main runs use 2,000 prompts per domain, 6,000 in total, which is 1.0× the clients' 6,000 private problems. We vary the pool size under unbalanced domains (β=1\beta=1):

Table R3: FedOPD with different server prompt-pool sizes under unbalanced domains (β=1\beta=1; fine-grained math, 3B).

Pool sizePrompt/privateGSM-HMATH500Int-NTAVG
3,0000.5×45.9460.2060.7451.48
6,0001.0×45.4162.4061.3051.74
12,0002.0×46.2863.5062.6752.67
6,000, matched1.0×47.3162.2063.7052.51

With half the prompts and unbalanced domains, FedOPD (51.48) still exceeds the best federated baseline (51.34). Doubling the pool offsets the imbalance (52.67 vs. 52.51 matched), mainly because the under-represented domains then have more distinct prompts.

W2. Baselines and code. Appendix. A.2 already describes each baseline and its upload. All methods share the student initialization, prompts, client experts, domain order, and step budget (hyperparameters in Appendix. A.1, experts in Appendix. B.3). Appendix A will add the shared learning-rate schedule and a per-baseline hyperparameter table. We will open-source the full code with the camera-ready version.

Dear Reviewer N5ZG,

We are writing to kindly ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

Reviewer_N5ZGOpenReview ↗

Rebuttal Assessment

W1/Q1 – Server prompts. The assumption is clarified, and privacy-filtering experiments are appreciated, although obtaining representative prompts remains a practical challenge. I consider this partially addressed.

Q2 – Distribution mismatch. The new experiments demonstrate robustness under moderate mismatch and acknowledge limitations under extreme mismatch. I consider this addressed.

Q3 – Prompt budget. The additional experiments are relevant, although the minimum prompt budget remains substantial and improvements over baselines become marginal. I consider this mainly addressed for what I asked.

W2 – Reproducibility: Implementation details are provided but (as far as I can see) reproducible code is not yet available. Reproducibility cannot be verified.

Overall, I appreciate the authors' efforts and the targeted experiments, which substantially mitigate my concerns. Some limitations remain, particularly regarding generalizability beyond the 3B fine-grained math setting for which most of the additional results are provided. Nevertheless, I have updated my score to weak accept in recognition of the improvements.

Reviewer_r5aGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a new federated "on-policy distillation" method for foundation models, when the clients participating in federated learning have heterogeneous probability distributions and participate in the learning process sequentially. They propose FedOPD, a general-purpose framework that is generation free, and SBPO, a domain incremental learning technique to solve the sequential scenario. The authors validate their proposed methods using open source large language models (Qwen family) on a variaty of multi-domain scenarios.

Critical strengths
  • The paper is a nice intersection of very important problems in the current machine learning landscape, among which: federated learning, domain incremental learning, and the fune-tuning of foundation models
  • Experiments are comprehensive and span a wide range of domains (e.g., math, code, science, medicine, finance).
Critical weaknesses

The main problems with this paper are clarity, and positioning.

Concerning Clarity.

Figure 1 contains a few problems, and it seems to conflict with algorithm 1. For instance, there is an arrow directed from clients (esp. client 1) towards the server, indicating that prompt xx is sent from clients to the server student. Nonetheless it is assumed that the server holds unlabeled prompt sets { Xk }k=1K\set{\mathcal{X}_{k}}_{k=1}^{K} (c.f. Section 3.1, line 146), contradicting Figure 1.

On top of that, there is no clear partitioning between cleint and server. Indeed, one would imagine that the central part of that figure would define the server and what it does, but it does not, since "client teacher" is part of that subfigure. Overall, I think the authors should rework Figure 1 to better illustrate their method, especially Algorithm 1.

Concerning Positioning.

From the introduction and related work, the authors seem to position their paper as a federated learning algorithm for foundation models, under a specific order of participation of clients. This particular order seems to be domain incremental learning, that is, one domain arrives after the other, since each client is identified with a different domain (e.g. client 1 -> math, cleint 2 -> science, client 3 -> medicine). Even the "sequential distillation" experiments test methods from the incremental learning community, such as Elastic Weight Consolidation (EWC; ; named "Fisher" in the paper) and Learning without Forgetting (LwF; ). There are a few problems with this framing, however,

  1. The authors do not discuss nor justify the continual learning angle in the introduction or related work,

  2. The methods are named without a proper citation,

While (2) is easily fixable (just add the citations below), (1) seems to be more complicated as the authours would need to clarify the precise setting they are proposing to solve in the introduction, and effectively report metrics such as stability and plasticity over the continual learning process.

References

Note. All of the references below are guess-work from the usual methods used in the continual learning literature... The authors should have properly cited the work below,

Fisher, which I am assuming it is EWC

Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., & Hadsell, R. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13), 3521–3526. https://doi.org/10.1073/pnas.1611835114

Learning without Forgetting (LwF)

Li, Z., & Hoiem, D. (2018). Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12), 2935–2947. https://doi.org/10.1109/TPAMI.2017.2773081

Memory Aware Synapses (MAS)

Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., & Tuytelaars, T. (2018). Memory aware synapses: Learning what (not) to forget. In V. Ferrari, M. Hebert, C. Sminchisescu, & Y. Weiss (Eds.), Computer Vision – ECCV 2018 (Lecture Notes in Computer Science, Vol. 11207, pp. 144–161). Springer. https://doi.org/10.1007/978-3-030-01219-9_9

Synaptic Intelligence (SI)

Zenke, F., Poole, B., & Ganguli, S. (2017). Continual learning through synaptic intelligence. In D. Precup & Y. W. Teh (Eds.), Proceedings of the 34th International Conference on Machine Learning (Vol. 70, pp. 3987–3995). PMLR. https://proceedings.mlr.press/v70/zenke17a.html

Questions for authors

N/A

I will gladly raise my scores if the authors solve the problems raised in the weaknesses section.

2 comments
AuthorsOpenReview ↗

Separate server and client roles in Figure 1 and clarify our core contribution, with SPBO handling sequential clients with both stability and plasticity

We thank the reviewer for the concrete suggestions. All revisions below will appear in the final version.

Figure 1. For a clearer presentation, we have revised Figure 1:

  • The frozen client teacher in the central panel is now drawn in the client color, separate from the server's operations, so the server–client boundary follows Algorithm 1 (steps 5–8).
  • The arrow from a client to xx denotes the problem text of that client's domain, with answers removed. As already described in App. B.1, this text forms the server's unlabeled pool XkX_k. This is consistent with Sec. 3.1: the server holds only unlabeled prompts and never receives answers, parameters, or generated content.

Positioning. Our contribution is the novel transfer protocol. Clients act as passive token-level critics and upload no parameters, no generated content, and no outcome-based signal. Sequential availability is a deployment constraint: clients are rarely online together.

As the reviewer notes, the resulting distillation is domain-incremental; this is why we added SPBO, which handles it without changing the protocol. Sec. 3.1 (lines 178–180) already notes that C1 turns the objective into a continual-learning problem; we agree this angle must be made explicit earlier. In the final version we will:

  1. Motivate sequential availability (C1, Sec. 3.1) in the introduction with real deployments, e.g., hospitals or banks that join the federation at different times or are reachable only in their own maintenance windows. We will state precisely the setting it induces: one domain per client, no replay of earlier prompts or feedback.
  2. Add a continual-learning paragraph to Related Work. It will cover regularization methods (EWC, SI, MAS), distillation-based LwF, and the on-policy forgetting results we already cite.

Stability and plasticity. We evaluate stability and plasticity under the same cross-domain setting as Table 2 (3B). Math is learned first, so its final score measures stability; Science is learned last, so its score measures plasticity.

Table R1: Per-domain averages after sequential distillation, Math→Code→Science.

MethodMath (1st) ↑ stabilityCode (2nd)Science (last) ↑ plasticityAVG
Student base48.1047.6443.2746.34
Global cosine48.3347.9743.9346.74
Blockwise LR51.6948.9344.5748.39
EWC (Kirkpatrick et al., 2017)50.6547.6442.6246.97
MAS (Aljundi et al., 2018)52.5550.1443.8048.83
LwF (Li & Hoiem, 2018)51.9350.4144.4848.94
SI (Zenke et al., 2017)54.2450.6844.5549.83
FedOPD with SPBO54.5150.6846.9750.72
  • Stability. Under the global cosine schedule, Math falls from 51.01 after its own block to 48.33 at the end (see Table 7). With SPBO it ends at 54.51. With five sequential clients (see Fig. 2c), each earlier domain moves by about one point or less.
  • Plasticity. Science, learned last, gains only +0.66 over the base under the global schedule, versus +3.70 with SPBO.
  • Order. Across all six Math/Code/Science orders, the macro average stays within 49.87–50.72 (see Table 9).

Citations. Thank you for the references.

  • SI was already cited where SPBO is defined (Zenke et al., 2017; line 231). Its link to Fisher-based importance was already cited in App. D.3 (Benzing, 2022).
  • "Fisher-FedOPD" denotes EWC. In the final version, Sec. 4.3 will state this and cite Kirkpatrick et al. (2017), together with LwF (Li & Hoiem, 2018) and MAS (Aljundi et al., 2018), which serve as baselines in Tables 2 and 7.
  • Related Work will cite all four methods.

References

  • Kirkpatrick, J., Pascanu, R., Rabinowitz, N., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017.
  • Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, T. Memory aware synapses: Learning what (not) to forget. In ECCV, LNCS 11207, pp. 144–161, 2018.
  • Li, Z. and Hoiem, D. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935–2947, 2018.
  • Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence. In ICML, PMLR 70:3987–3995, 2017.
  • Benzing, F. Unifying importance based regularisation methods for continual learning. In AISTATS, PMLR 151, 2022.

Dear Reviewer r5aG,

We are writing to kindly ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

Reviewer_SouGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates knowledge transfer in federated foundation models (FedFMs), where a server-side foundation model is enriched with domain expertise held by distributed clients in the form of locally fine-tuned expert models. To provide a privacy-preserving and communication-efficient client-to-server knowledge transfer, this paper presents FedOPD, an on-policy distillation protocol where the server generates reasoning trajectories and sends them to domain-matched clients. Frozen client experts in FedOPD evaluate these trajectories through teacher forcing and return only the log-probability of each sampled token. The server uses the teacher–student log-probability gaps as token-level advantages in a PPO-style update. The paper provides both theoretical analysis and empirical results on FedOPD across two task settings of mathematical reasoning and cross-domain knowledge transfer, using Qwen2.5 students at two model scales.

Critical strengths
  • This paper aims to address an important and practical problem in knowledge transfer for FedFMs.

  • FedOPD is both parameter-free, generation-free, and outcome-free from the perspective of the clients, where no local-model-related sensitive information is reported to the server.

  • The experiments report consistent performance gains from FedOPD in average accuracy across both student sizes and task settings, while introducing lower communication overhead than the evaluated parameter-sharing and full-distribution distillation baselines.

Critical weaknesses
  • Since privacy is the main motivation for preferring FedOPD over FedDF and FedGRPO, the paper needs more empirical privacy-leakage comparisons to justify such a claim. Sepecifically, the paper should narrow its claims and evaluate information leakage under its stated semi-honest threat model.
  • The main comparisons do not clearly separate the contribution of the transfer protocol from that of SPBO. FedOPD includes SPBO by default, and Table 2 shows that SPBO improves cross-domain accuracy by a large margin. However, SPBO is an optimizer-level continual-learning component that is orthogonal to what clients upload and could be applied to every sequential baseline. Do the competing methods receive comparable learning-rate restarts and forgetting mitigation?
  • The experiments do not clearly isolate the value of client-specific knowledge. For instance, Table 6 shows that the pretrained teacher achieves a similar performance to the GRPO-trained client experts. In this context, it is unclear how much of the performance gain on the server-side student model comes from decentralized client-specific knowledge rather than knowledge already available in the pretrained teacher.
  • One assumption for the server in FedOPD is that it can access the problem texts from the local data silos, which naturally provides closely aligned transfer prompts, whereas such inputs may also be private or unavailable in the real-world deployment. It would be better to involve more discussion on this assumption.
  • The theoretical results provide narrower support than the approximations used in training. The shared fixed-point result is established for independently parameterized per-prefix logits, rather than the shared-parameter LoRA setting used in the experiments.
  • No standard deviations are reported despite three seeds, while many gaps are around one point on benchmarks.
Questions for authors

See the concerns in the Weakness section above.

2 comments
AuthorsOpenReview ↗

FedOPD leaks little under a semi-honest server and stays ahead when every baseline uses SPBO, with gaps well above seed variance

We thank the reviewer for the careful review. All new results below will appear in the revised version.

W1. Leakage under the semi-honest model. A semi-honest server runs a membership-inference attack (MIA) (Shokri et al., 2017; Nasr et al., 2019; Carlini et al., 2022) on each protocol's payload to infer whether a problem was in an expert's training set. We use 384 length-matched member/non-member test problems; attacker, features, seed and split are shared (revised Appendix C).

Table R1: MIA by a semi-honest server (%; TPR at fixed FPR).

MethodPayloadAUC↓ASR↓TPR@1%↓TPR@5%↓TPR@10%↓
FedGRPOrewards51.4953.130.000.007.81
FedOPDlog-probs53.6050.000.005.2116.67
FedDFtop-64 dist.60.1653.1310.4217.1926.56
FedPDPOpref. pairs63.5463.5427.0827.0827.08
DPSDA-FLsynthetic data98.4194.2771.8894.2796.35

FedOPD's AUC is within the ±5.8-point 95% band of chance and it finds no member at 1% FPR; FedDF/FedPDPO expose 10–27% of members, DPSDA-FL nearly all. FedGRPO leaks slightly less but needs private verifiers and loses 4.28 points. We will narrow the privacy claims accordingly and claim no formal guarantee.

References

  • Shokri, R., et al. Membership inference attacks against machine learning models. IEEE S&P, 2017.
  • Nasr, M., Shokri, R., and Houmansadr, A. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. IEEE S&P, 2019.
  • Carlini, N., et al. Membership inference attacks from first principles. IEEE S&P, 2022.

W2. SPBO vs. protocol. All methods in Table 1 already share the blockwise LR schedule. We add the forgetting-mitigation strategy SI to every baseline, so all use full SPBO (revised Appendix C).

Table R2: All methods with and without SI (math, 3B).

MethodMATH500Int-ALInt-GEInt-NTAVG w/o SIAVG +SPBO
FedOPD62.2081.5551.9863.7051.7552.51
FedDF61.2083.4050.3159.4451.3351.86
FedPDPO60.4082.3148.2358.8949.4351.51
FedGRPO61.0080.3750.1060.0048.2350.61

SPBO helps all methods (+0.53 to +2.38); FedOPD is best with and without SI.

W3. Client-specific knowledge. We add FedOPD with the pretrained teacher (revised Appendix C); expert scores are already in Table 6.

Table R3: Experts and their FedOPD students (math, 3B).

ExpertsExpert Int-AL/GE/NTExpert AVGStudent Int-ALInt-GEInt-NTMATH500Student AVG
SFT72.70/37.79/62.7853.10–57.8580.3749.0659.0759.8050.25
Pretrained85.52/63.47/76.7063.4181.8952.1961.8560.4052.09
GRPO87.19/64.51/77.9663.47–63.7981.5551.9863.7062.2052.51

GRPO gains concentrate on each expert's own sub-field. The student follows local training both ways (SFT −1.84, GRPO +0.42). Most of the gain over the base (46.81→52.09) is math expertise the 3B generalist lacks.

W4. Server-side prompts. FedMD, FedDF and FedGRPO share this assumption; it fits when value lies in answers, not questions (e.g., public question banks with proprietary solutions, already in App. B.1). Private problem text with personally identifiable information (PII) can be filtered before release (revised Appendix C).

Table R4: Privacy-filtered server prompts (math, 3B).

Server promptsPII releasedGSM-HMATH500Int-NTAVG
Originalyes47.3162.2063.7052.51
Remove PII problemsno46.2561.2060.9352.39
Mask PII spansmasked44.9661.0060.9351.69

Removing all PII-bearing problems (about 35%) costs only 0.12 points; masking costs 0.82, as placeholders make some problems ambiguous. For mismatched or scarce prompts, see our reply to Reviewer N5ZG.

W5. Theory under shared parameters. Only the "only if" in Prop. 1(i) uses per-prefix logits; all other results hold for any parameterization, LoRA or full. A new corollary (revised Appendix D) bounds the FedOPD term's gap to the exact ∇Jk\nabla J_k by 2GJ(T−1)KL(πθ∥πϕ)\sqrt2G_J(T-1)\mathrm{KL}(\pi_\theta\|\pi_\phi). A realizable teacher is still a shared fixed point, and any other fixed point is ε-stationary for JkJ_k with ε proportional to the residual KL. Only uniqueness is lost, as in policy-gradient theory beyond tabular softmax (Agarwal et al., 2021). Empirically, FedOPD lowers this residual KL in every domain under LoRA (Algebra 0.51→0.31, Geometry 0.63→0.35, Number Theory 0.32→0.16 nats/token; 64 test prompts each).

References

  • Agarwal, A., et al. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. JMLR, 2021.

W6. Std. Table 1 reports means for width; Table R6 adds std over three seeds (full table in revised Appendix C).

Table R6: AVG mean ± std over three seeds.

SettingFedOPDStrongest baselineGap
Math 3B52.51±0.1151.34±0.09 (DPSDA-FL)1.17
Math 7B61.63±0.0760.25±0.22 (FedDF)1.38
Cross 3B50.72±0.2448.69±0.08 (FedDF)2.03
Cross 7B58.27±0.2954.85±0.22 (FedPDPO)3.42

Dear Reviewer SouG,

We are writing to kindly ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief introduction to the revised PDF

We thank all reviewers for their time and constructive feedback.

New content is added after line 937, so that line references stay valid. The few text edits before it are listed at the end.

New experiments and analysis:

  • Table 13 (membership inference by a semi-honest server): On FedOPD's log-probabilities the attack stays within the chance band and finds no member at 1% FPR. FedDF and FedPDPO expose 10–27% of members, and DPSDA-FL nearly all (to Reviewer SouG).
  • Table 14 (mismatched and smaller server prompt pools): With moderately mismatched prompts or only half of the prompts, FedOPD still outperforms every baseline that uses matched prompts (to Reviewer N5ZG).
  • Table 15 (all baselines with full SPBO): FedOPD remains the best method, with about 1% of FedDF's upload (to Reviewer SouG).
  • Table 16 (pretrained vs. locally trained teachers): FedOPD transfers math expertise that the 3B generalist lacks (46.81→52.51). The student also improves with the experts' local training: GRPO experts beat the pretrained teacher, and SFT experts do worse than both (to Reviewer SouG).
  • Table 17 (privacy-filtered server prompts): Removing every prompt that contains personally identifiable information costs only 0.12 points (to Reviewers SouG and N5ZG).
  • Corollary 1 (line 1194): The FedOPD theory extends to shared parameters such as LoRA. A realizable teacher remains a fixed point, and any other fixed point is near-stationary up to the residual KL (to Reviewer SouG).

Text revisions:

  • Figure 1 is redrawn to separate client-side and server-side operations (to Reviewer r5aG).
  • Sec. 4.3 cites EWC (the "Fisher" variant), MAS, LwF, and SI (to Reviewer r5aG).
  • The privacy wording in the abstract and introduction is narrowed (to Reviewer SouG).
  • Sec. 3.1, Sec. 4.1 and App. A.2 add short pointers to the new results. App. A.2 also states that all baselines share the blockwise learning-rate schedule (to Reviewers SouG and N5ZG).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.