- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies federated on-policy distillation, where clients with disjoint prompt sets share a fixed teacher and aggregate LoRA updates with FedAvg. The authors observe that FedAvg gives almost no gain over local training at a learning rate of 3e-6 but a clear gain at 9e-6. They explain this with a solvable model in which aggregation slows progress, so the student reaches useful teaching states less often in its own rollouts. They then propose FedTOPS, which picks a server multiplier for the FedAvg update using teacher KL on cached prefixes under a per-client KL constraint. On six math benchmarks, it improves Macro Avg@8 over FedAvg by 4.56 to 14.57 points.
Critical strengths
- The observation that the student is both the learner and the data generator in federated OPD is interesting, and the toy model makes it concrete. Theorem 1 isolates the rollout-mediated cost, and Corollary 4 shows that it vanishes when supervision access is fixed. I was able to reproduce Table 4 from the parameters in Appendix C.7.
- FedTOPS is simple and cheap because it reuses the training rollouts (8-15% overhead, Table 10), and its approximations are clearly documented in Appendices D and E.
Critical weaknesses
- My main concern is that the experiments do not rule out a simpler explanation for the central observation. The paper attributes the missing collaboration gain at 1xLR to aggregation delaying access to teaching states, but it could also be that the step is too small for any method to learn much within the budget. At 1xLR all non-centralized methods stay close to the base model (Macro Avg@8 of 3.91-4.13 vs. 3.32). Across Table 1, the scores also seem to track the effective step no matter whether it comes from the local learning rate or the server multiplier, for example 16.19 for at 1xLR and 16.40 for at 3xLR. Both explanations predict that larger steps help, so these results cannot distinguish them. The direct evidence for the mechanism is also limited. Centralized uses 3x more sequential steps than FedAvg (Appendix F.2), so the gap between them is not only due to aggregation. The intervention in Section 4.4 shows that a better rollout source helps later learning, but it does not show that this is what causes the learning rate dependence.
- The benefit of the teacher-guided selection over a constant server multiplier is not established. In Table 1, fixed reaches 16.19 vs. 16.84 for FedTOPS at 1xLR, and fixed reaches 16.40 vs. 17.33 at 3xLR, recovering about 95% and 80% of FedTOPS's gain over FedAvg. With a single seed, and with fixed scaling slightly ahead on Pass@8 (Table 2), the remaining gap is difficult to interpret. Table 8 points the same way: at 1xLR the KL constraint never excludes a candidate and is selected in 37 of 50 rounds, so FedTOPS effectively acts as a fixed multiplier. Its remaining advantage, avoiding retuning when the best fixed a shifts from 10 to 3, could also come from a teacher-free adaptive server step such as FedExP[1], which the paper cites but does not compare with. The 1.7B experiments also omit the fixed-scaling baselines.
[1] D. Jhunjhunwala, S. Wang, and G. Joshi. FedExP: Speeding Up Federated Averaging via Extrapolation. ICLR 2023.
Questions for authors
- Why does this problem need federation? If each client runs the 8B teacher locally, the other clients only add prompts, so would local OPD on public or synthetic math prompts do as well as FedAvg? In-domain results for each client, and a concrete scenario where prompts are private but every client can run an 8B teacher, would make the case much stronger.
- Regarding W1, could you repeat the FedAvg vs. Local comparison at 1xLR with rollouts generated once by the initial student and kept fixed? Following Corollary 4, the proposed mechanism predicts a positive gain in this case, while a gain near zero would point to the step size instead.
- Regarding W2, could you compare FedTOPS with a tuned constant server step and FedExP, add the fixed baselines for the 1.7B student, and report Local and FedAvg at a larger learning rate such as 3e-5, over a few seeds? If FedTOPS keeps a clear margin, I would raise my score.
- How does FedTOPS behave without the KL constraint or with other values of ?
15 comments
Response to Q1 (Part 1 / 2)
Q1: Why does this problem need federation? If each client runs the teacher locally, the other clients only add prompts, so would local OPD on public or synthetic math prompts do as well as FedAvg? In-domain results for each client, and a concrete scenario where prompts are private but every client can run an teacher, would make the case much stronger.
A1: Thank you for this valuable question. FedOPD targets the distillation of a shared large teacher into a compact student for private, domain-specific applications on resource-constrained devices. Institutions use the teacher during training, while federation allows the student to learn from their complementary private task data without pooling raw prompts or trajectories. Our domain-level evaluation shows that FedTOPS improves over the corresponding Local model on all client-aligned domains across all reported settings.
Practical Motivation:
Consider financial institutions distilling a compact risk-analysis model on confidential transaction cases. Each institution can run a local copy of the frozen Qwen3-8B teacher on its training server and deploy the smaller student on resource-constrained devices. Federation combines student LoRA updates across institutions while keeping private prompts and generated trajectories local.
Public and Synthetic Prompts:
We focus on private, specialized tasks for which suitable public prompts are scarce and generic synthetic prompts may not capture the relevant business context. In OPD, the teacher supervises the current student's own rollouts. Mismatched prompts can therefore direct teacher feedback toward reasoning situations that differ from those encountered in the intended application. The private prompts contributed by each client provide the task context needed to make this supervision relevant.
Following common practice in OPD for reasoning, we use mathematical benchmarks as a controlled testbed for collaboration across distinct client task distributions.
Client-Specific Results:
We partitioned MATH500 using the same subject taxonomy as the training clients:
- C1: Prealgebra and Algebra.
- C2: Number Theory and Counting and Probability.
- C3: Intermediate Algebra and Precalculus.
Geometry is not assigned to any training client and is reported as a held-out subject. For each client-aligned subset, the domain-matched Local baseline uses that client's own trained model, allowing a direct comparison with the global student on the client's task domain.
Response to Q1 (2 / 2)
R3-Table1: Domain-level MATH500 (%). Question counts are shown in parentheses.
For Local (domain-matched), each client-aligned subset is evaluated using its corresponding client model; Geometry averages all local models. For each global model, the overall MATH500 score is the question-count-weighted average across the subsets. Bold and underlined values indicate the highest and second-highest scores in each column.
FedTOPS exceeds both the domain-matched Local baseline and FedAvg on every client-aligned domain across the reported settings. At , its gains over the corresponding Local models are , , and percentage points for C1, C2, and C3, respectively. Positive gains persist at , where the corresponding improvements are , , and points.
These improvements occur within each client's own subject areas, directly supporting the value of collaborative distillation for the participating clients. FedTOPS also exceeds Local and FedAvg on the held-out Geometry subject in every reported setting.
We hope this concrete deployment scenario and the domain-matched comparisons address your concern about the motivation and practical benefits of collaboration with a shared teacher.
Overview of Our Responses
We sincerely thank you for your careful review and constructive suggestions. We have expanded our evaluation with comparisons across multiple training seeds, model sizes, and learning rates, together with client-specific evaluations and controlled rollout-source experiments. Our responses follow your four Questions for Author, with W1 addressed in Q2 and W2 in Q3.
We have also completed rollout-source interventions at different training stages, a frozen-factor LoRA control, and sensitivity studies of candidate sets and scoring-cache configurations. These complementary studies are reported in our responses to Reviewer CuVx, should you wish to consult them.