acceptodds
Under review as a conference paper at ICLR 2027

Trapped by Their Own Rollouts: Understanding Aggregation–Rollout Feedback in Federated On-Policy Distillation

Abstract

On-policy distillation (OPD) is a promising approach to language-model adaptation, aligning teacher supervision with the student's own generated trajectories. When adaptation prompts are distributed across clients, can this process benefit from federated collaboration? We study federated OPD and find that substantial collaboration gains can be obscured by learning-rate sensitivity: FedAvg can perform no better than independent local training at a small learning rate, yet recover a clear advantage at a larger rate. We explain this phenomenon through the student's dual role as learner and generator of future training data. An aggregation-induced optimization lag can delay access to useful teacher supervision, which in turn slows subsequent learning. Our theory establishes this *aggregation–rollout feedback* in a solvable model with a common optimum and stable updates, and identifies two coupled roles of learning rate: learning from current supervision and reaching future supervision. Guided by this analysis, we propose FedTOPS (Federated Teacher-guided On-Policy Scaling), which reuses teacher feedback on current trajectories to adapt the FedAvg update magnitude under clientwise predictive-change constraints. Across six mathematical reasoning benchmarks, FedTOPS improves macro Avg@8 over FedAvg by 4.56–14.57 percentage points across the evaluated student models and local learning rates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Federated learning, on-policy distillation, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XJxUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies federated on-policy distillation, where clients with disjoint prompt sets share a fixed teacher and aggregate LoRA updates with FedAvg. The authors observe that FedAvg gives almost no gain over local training at a learning rate of 3e-6 but a clear gain at 9e-6. They explain this with a solvable model in which aggregation slows progress, so the student reaches useful teaching states less often in its own rollouts. They then propose FedTOPS, which picks a server multiplier for the FedAvg update using teacher KL on cached prefixes under a per-client KL constraint. On six math benchmarks, it improves Macro Avg@8 over FedAvg by 4.56 to 14.57 points.

Critical strengths
  1. The observation that the student is both the learner and the data generator in federated OPD is interesting, and the toy model makes it concrete. Theorem 1 isolates the rollout-mediated cost, and Corollary 4 shows that it vanishes when supervision access is fixed. I was able to reproduce Table 4 from the parameters in Appendix C.7.
  2. FedTOPS is simple and cheap because it reuses the training rollouts (8-15% overhead, Table 10), and its approximations are clearly documented in Appendices D and E.
Critical weaknesses
  1. My main concern is that the experiments do not rule out a simpler explanation for the central observation. The paper attributes the missing collaboration gain at 1xLR to aggregation delaying access to teaching states, but it could also be that the step is too small for any method to learn much within the budget. At 1xLR all non-centralized methods stay close to the base model (Macro Avg@8 of 3.91-4.13 vs. 3.32). Across Table 1, the scores also seem to track the effective step α⋅η\alpha\cdot\eta no matter whether it comes from the local learning rate or the server multiplier, for example 16.19 for a=10a=10 at 1xLR and 16.40 for a=3a=3 at 3xLR. Both explanations predict that larger steps help, so these results cannot distinguish them. The direct evidence for the mechanism is also limited. Centralized uses 3x more sequential steps than FedAvg (Appendix F.2), so the gap between them is not only due to aggregation. The intervention in Section 4.4 shows that a better rollout source helps later learning, but it does not show that this is what causes the learning rate dependence.
  2. The benefit of the teacher-guided selection over a constant server multiplier is not established. In Table 1, fixed a=10a=10 reaches 16.19 vs. 16.84 for FedTOPS at 1xLR, and fixed a=3a=3 reaches 16.40 vs. 17.33 at 3xLR, recovering about 95% and 80% of FedTOPS's gain over FedAvg. With a single seed, and with fixed scaling slightly ahead on Pass@8 (Table 2), the remaining gap is difficult to interpret. Table 8 points the same way: at 1xLR the KL constraint never excludes a candidate and a=10a=10 is selected in 37 of 50 rounds, so FedTOPS effectively acts as a fixed multiplier. Its remaining advantage, avoiding retuning when the best fixed a shifts from 10 to 3, could also come from a teacher-free adaptive server step such as FedExP[1], which the paper cites but does not compare with. The 1.7B experiments also omit the fixed-scaling baselines.

[1] D. Jhunjhunwala, S. Wang, and G. Joshi. FedExP: Speeding Up Federated Averaging via Extrapolation. ICLR 2023.

Questions for authors
  1. Why does this problem need federation? If each client runs the 8B teacher locally, the other clients only add prompts, so would local OPD on public or synthetic math prompts do as well as FedAvg? In-domain results for each client, and a concrete scenario where prompts are private but every client can run an 8B teacher, would make the case much stronger.
  2. Regarding W1, could you repeat the FedAvg vs. Local comparison at 1xLR with rollouts generated once by the initial student and kept fixed? Following Corollary 4, the proposed mechanism predicts a positive gain in this case, while a gain near zero would point to the step size instead.
  3. Regarding W2, could you compare FedTOPS with a tuned constant server step and FedExP, add the fixed baselines for the 1.7B student, and report Local and FedAvg at a larger learning rate such as 3e-5, over a few seeds? If FedTOPS keeps a clear margin, I would raise my score.
  4. How does FedTOPS behave without the KL constraint or with other values of δ\delta?
15 comments
AuthorsOpenReview ↗

Overview of Our Responses

We sincerely thank you for your careful review and constructive suggestions. We have expanded our evaluation with comparisons across multiple training seeds, model sizes, and learning rates, together with client-specific evaluations and controlled rollout-source experiments. Our responses follow your four Questions for Author, with W1 addressed in Q2 and W2 in Q3.

We have also completed rollout-source interventions at different training stages, a frozen-factor LoRA control, and sensitivity studies of candidate sets and scoring-cache configurations. These complementary studies are reported in our responses to Reviewer CuVx, should you wish to consult them.

AuthorsOpenReview ↗

Response to Q1 (Part 1 / 2)

Q1: Why does this problem need federation? If each client runs the 8B8\mathrm{B} teacher locally, the other clients only add prompts, so would local OPD on public or synthetic math prompts do as well as FedAvg? In-domain results for each client, and a concrete scenario where prompts are private but every client can run an 8B8\mathrm{B} teacher, would make the case much stronger.

A1: Thank you for this valuable question. FedOPD targets the distillation of a shared large teacher into a compact student for private, domain-specific applications on resource-constrained devices. Institutions use the teacher during training, while federation allows the student to learn from their complementary private task data without pooling raw prompts or trajectories. Our domain-level evaluation shows that FedTOPS improves over the corresponding Local model on all 33 client-aligned domains across all 44 reported settings.

Practical Motivation:

Consider financial institutions distilling a compact risk-analysis model on confidential transaction cases. Each institution can run a local copy of the frozen Qwen3-8B teacher on its training server and deploy the smaller student on resource-constrained devices. Federation combines student LoRA updates across institutions while keeping private prompts and generated trajectories local.

Public and Synthetic Prompts:

We focus on private, specialized tasks for which suitable public prompts are scarce and generic synthetic prompts may not capture the relevant business context. In OPD, the teacher supervises the current student's own rollouts. Mismatched prompts can therefore direct teacher feedback toward reasoning situations that differ from those encountered in the intended application. The private prompts contributed by each client provide the task context needed to make this supervision relevant.

Following common practice in OPD for reasoning, we use mathematical benchmarks as a controlled testbed for collaboration across distinct client task distributions.

Client-Specific Results:

We partitioned MATH500 using the same subject taxonomy as the training clients:

  • C1: Prealgebra and Algebra.
  • C2: Number Theory and Counting and Probability.
  • C3: Intermediate Algebra and Precalculus.

Geometry is not assigned to any training client and is reported as a held-out subject. For each client-aligned subset, the domain-matched Local baseline uses that client's own trained model, allowing a direct comparison with the global student on the client's task domain.

AuthorsOpenReview ↗

Response to Q1 (2 / 2)

R3-Table1: Domain-level MATH500 Avg@8\mathrm{Avg}@8 (%). Question counts are shown in parentheses.

0.6B/1×LR0.6\mathrm{B}/1\times\mathrm{LR}

Method\mathrm{Method}C1;(206)\mathrm{C1}\\;(206)C2;(100)\mathrm{C2}\\;(100)C3;(153)\mathrm{C3}\\;(153)Geometry;(41)\mathrm{Geometry}\\;(41)
Local;(domain-matched)\mathrm{Local}\\;(\text{domain-matched})19.1719.176.256.257.197.1914.0214.02
FedAvg\mathrm{FedAvg}17.9617.966.006.007.277.2711.2811.28
TunedFixed\mathrm{Tuned\\ Fixed}62.62‾\underline{62.62}37.88‾\underline{37.88}24.75‾\underline{24.75}38.72\mathbf{38.72}
FedTOPS\mathrm{FedTOPS}64.50\mathbf{64.50}39.50\mathbf{39.50}27.29\mathbf{27.29}38.41‾\underline{38.41}

0.6B/3×LR0.6\mathrm{B}/3\times\mathrm{LR}

Method\mathrm{Method}C1;(206)\mathrm{C1}\\;(206)C2;(100)\mathrm{C2}\\;(100)C3;(153)\mathrm{C3}\\;(153)Geometry;(41)\mathrm{Geometry}\\;(41)
Local;(domain-matched)\mathrm{Local}\\;(\text{domain-matched})47.2147.2116.7516.7514.0514.0528.8628.86
FedAvg\mathrm{FedAvg}51.9451.9431.6231.6221.4921.4936.2836.28
TunedFixed\mathrm{Tuned\\ Fixed}64.99‾\underline{64.99}38.00‾\underline{38.00}27.29\mathbf{27.29}39.94‾\underline{39.94}
FedTOPS\mathrm{FedTOPS}66.50\mathbf{66.50}41.75\mathbf{41.75}26.88‾\underline{26.88}45.43\mathbf{45.43}

0.6B/10×LR0.6\mathrm{B}/10\times\mathrm{LR}

Method\mathrm{Method}C1;(206)\mathrm{C1}\\;(206)C2;(100)\mathrm{C2}\\;(100)C3;(153)\mathrm{C3}\\;(153)Geometry;(41)\mathrm{Geometry}\\;(41)
Local;(domain-matched)\mathrm{Local}\\;(\text{domain-matched})64.08‾\underline{64.08}36.2536.2525.4125.4140.65‾\underline{40.65}
FedAvg\mathrm{FedAvg}63.5963.5939.38‾\underline{39.38}26.96‾\underline{26.96}40.5540.55
FedTOPS\mathrm{FedTOPS}67.11\mathbf{67.11}44.50\mathbf{44.50}28.51\mathbf{28.51}41.77\mathbf{41.77}

1.7B/1×LR1.7\mathrm{B}/1\times\mathrm{LR}

Method\mathrm{Method}C1;(206)\mathrm{C1}\\;(206)C2;(100)\mathrm{C2}\\;(100)C3;(153)\mathrm{C3}\\;(153)Geometry;(41)\mathrm{Geometry}\\;(41)
Local;(domain-matched)\mathrm{Local}\\;(\text{domain-matched})39.2039.2027.3827.3819.7719.7722.1522.15
FedAvg\mathrm{FedAvg}37.9237.9225.6225.6220.6720.6721.3421.34
TunedFixed\mathrm{Tuned\\ Fixed}77.25‾\underline{77.25}61.88\mathbf{61.88}42.97‾\underline{42.97}50.61‾\underline{50.61}
FedTOPS\mathrm{FedTOPS}78.16\mathbf{78.16}61.38‾\underline{61.38}43.87\mathbf{43.87}54.57\mathbf{54.57}

For Local (domain-matched), each client-aligned subset is evaluated using its corresponding client model; Geometry averages all 33 local models. For each global model, the overall MATH500 score is the question-count-weighted average across the 44 subsets. Bold and underlined values indicate the highest and second-highest scores in each column.

FedTOPS exceeds both the domain-matched Local baseline and FedAvg on every client-aligned domain across the reported settings. At 0.6B/1×LR0.6\mathrm{B}/1\times\mathrm{LR}, its gains over the corresponding Local models are 45.3345.33, 33.2533.25, and 20.1020.10 percentage points for C1, C2, and C3, respectively. Positive gains persist at 10×LR10\times\mathrm{LR}, where the corresponding improvements are 3.033.03, 8.258.25, and 3.103.10 points.

These improvements occur within each client's own subject areas, directly supporting the value of collaborative distillation for the participating clients. FedTOPS also exceeds Local and FedAvg on the held-out Geometry subject in every reported setting.

We hope this concrete deployment scenario and the domain-matched comparisons address your concern about the motivation and practical benefits of collaboration with a shared teacher.

Reviewer_pPY1OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates learning-rate sensitivity in federated on-policy distillation and explains it through aggregation–rollout feedback. A simple teacher-guided scaling method, FedTOPS, is proposed. I find the insight interesting and the results convincing.

Critical strengths
  1. The key observation is insightful. Exploring OPD in federated training is also very interesting. This paper shows that OPD can achieve improvement in decentralized training but have pitfall. It also shows that aggregation changes not only the model, but also what it can learn from its future rollouts.
  2. The theory is well motivated. The source-swapping experiment provides useful support for the proposed explanation.
  3. The method is simple and effective. FedTOPS clearly improve the performance of FedOPD in relatively low lr scenarios, making it insensitive to lr choices.
Critical weaknesses
  1. From an FL perspective, the relation between the proposed optimization lag and the usual client drift is not very clear to me. Some discussion would help.
  2. The practical setup is not clear. All clients use a shared teacher. Is the teacher deployed at each client or accessed from the server?
  3. Candidate scoring needs evaluation on clients. Does this require extra model communication within each round? The overhead should be discussed.
Questions for authors

See Weaknesses.

8 comments
AuthorsOpenReview ↗

Overview of Our Responses

We sincerely thank you for your thoughtful and constructive review. We appreciate your recognition of the student's dual role as learner and rollout generator, our theoretical motivation, and the effectiveness of FedTOPS. Below, we clarify the relation between optimization lag and client drift, explain how the teacher is deployed, and detail the communication and runtime costs of candidate selection.

We have also substantially expanded our evaluation across multiple training seeds and added controlled experiments on aggregation–rollout feedback. These results strengthen the evidence for our main claims and are presented in our responses to the other reviewers, should you wish to consult them.

AuthorsOpenReview ↗

Response to Q1

Q1: From an FL perspective, the relation between the proposed optimization lag and the usual client drift is not very clear to me. Some discussion would help.

A1: Thank you for raising this point. The optimization lag in our analysis is connected to the usual discrepancy caused by heterogeneous local updates. Our central insight is that, in FedOPD, this lag can also delay access to useful teacher supervision because the student generates the trajectories used in subsequent training.

Relation to Client Drift:

With heterogeneous client objectives, multiple local steps can move clients along different update paths. Averaging their models can therefore differ from a synchronous pooled reference that averages client gradients at every step. Proposition 22, Eq. (10)(10), characterizes this familiar discrepancy with the rollout source held fixed. In our construction, it slows the aggregated model's progress relative to that reference, producing the optimization lag.

Additional Feedback through Rollouts:

In FedOPD, the aggregated student also generates the next round's training trajectories. In our construction, slower acquisition of a prerequisite ability makes the student visit useful teaching states for a downstream ability less often. It consequently receives less relevant supervision, further slowing its learning. Theorem 11 isolates this additional delay even when all clients share the same optimum and all updates remain stable.

Eq. (11)(11) separates the two propagation paths. Let δt\delta_t denote the parameter difference between federated training and the synchronous pooled reference. Its local propagation is

=(I-E\eta H_t)\delta_t+e_t -E\eta B_t\delta_t +\text{higher-order terms}.$$ Here, $E$ is the number of local steps, $\eta$ is the learning rate, and $e_t$ is the one-round aggregation discrepancy. The $H_t$ term describes propagation through learning with a fixed source. The $B_t$ term captures the additional effect of changing the rollout distribution on the subsequent training gradient. With a common externally fixed rollout source, the first-order $B_t$ contribution disappears, while ordinary optimization discrepancies can remain. This separates the usual effect of heterogeneous local optimization from the additional feedback through generated supervision. We will clarify this relationship in the main text. We hope this explanation addresses your concern about the connection between optimization lag and client drift.
AuthorsOpenReview ↗

Response to Q2

Q2: The practical setup is not clear. All clients use a shared teacher. Is the teacher deployed at each client or accessed from the server?

A2: Thank you for asking us to clarify the setup. In our experiments, the 33 logical clients and teacher inference run on a single RTX PRO 60006000 Blackwell GPU with 96,GB96\\,\mathrm{GB} memory. Each client uses the same frozen Qwen3-8B teacher to supervise its own student-generated trajectories. Here, “shared” refers to identical teacher parameters across clients. The server aggregates the student's LoRA updates, and the teacher remains fixed.

Our intended application is collaborative distillation across institutions with private domain-specific tasks. For example, financial institutions may seek a compact model for risk analysis using confidential transaction cases and internal business context. Each institution can host a local copy of the same open-weight teacher on its training infrastructure. Federated training combines the student updates while keeping raw prompts and trajectories within each institution. The resulting student can then be deployed on resource-constrained devices.

Although the teacher is shared, each institution contributes different task inputs and learning signals. Federation allows the student to learn from these complementary private task distributions. Following common practice in OPD for reasoning, we evaluate on public mathematical benchmarks, including MATH500. We partition the MATH training data by subject area across clients, providing a controlled setting for studying distillation across heterogeneous task distributions.

We will clarify the experimental implementation and intended deployment in the revised manuscript. We hope this explanation resolves your concern about teacher access and makes the practical setting clearer.

Reviewer_CuVxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates aggregation–rollout feedback in federated on-policy distillation, where aggregation-induced optimization lag can delay access to subsequent teacher supervision. The authors develop a theoretical construction and propose FedTOPS, which selects the aggregated update magnitude using cached teacher feedback and clientwise predictive-change constraints. Experiments on six mathematical reasoning benchmarks demonstrate improvements over FedAvg.

Critical strengths
  1. This paper is well-written and easy to follow. The distinction between the student’s roles as learner and trajectory generator clearly motivates the problem.
  2. The theoretical construction isolates aggregation–rollout feedback under a shared optimum and stable updates. Together with the rollout-source intervention, it provides useful evidence for a mechanism beyond ordinary optimization effects.
Critical weaknesses
  1. The additional advantage over simpler server-step adjustments is not sufficiently established, which limits the evidence for the proposed adaptive selector.
  2. Candidate selection introduces additional computation, but the performance–cost trade-off and the necessity of evaluating six candidates every round remain unclear.
  3. The experiments provide limited evidence regarding training variability, mechanism attribution, and sensitivity to key implementation choices.
Questions for authors
  1. In Table 1, FedTOPS improves macro Avg@8 over the stronger reported fixed-scaling baseline by only 0.65 and 0.93 percentage points at the two learning rates. These fixed rules also achieve slightly higher macro Pass@8. How does FedTOPS compare with more thoroughly tuned fixed multipliers and FedExP?
  2. Appendix G.3 reports auxiliary timing, but these measurements do not establish the end-to-end slowdown relative to FedAvg or fixed scaling. Could the authors provide accuracy-versus-wall-clock-time comparisons?
  3. What motivates the candidate set {0.5,1,2,3,5,10}\{0.5,1,2,3,5,10\}? Could smaller candidate sets retain similar accuracy with fewer forward evaluations? Since a=10a=10 is selected in 37 of 50 rounds at the reference learning rate, it would also be useful to examine whether frequent selection of the largest candidate indicates a restrictive search range.
  4. Every configuration uses one training seed. Do the improvements over the stronger fixed-scaling baselines persist across independent runs?
  5. The rollout-source intervention in Section 4.4 is useful, but considers only one checkpoint and a short continuation. Does the source advantage persist at other checkpoints? How much of the performance gap remains when different aggregation rules use a common externally fixed rollout source?
  6. Aggregation and scaling operate directly on both LoRA factors. As shown in Eq. (80), the effective weight update contains a quadratic term in the multiplier. Do the main observations persist with corrected LoRA aggregation or a frozen-factor parameterization?
  7. Each client uses four cached trajectories and at most 16 positions per trajectory, with compressed vocabulary distributions. How sensitive are candidate rankings and final accuracy to these choices? Since compressed KL does not certify the full-vocabulary constraint, how often do selected candidates exceed the corresponding full-vocabulary KL budget on an independent diagnostic sample?
  8. How much does the clientwise constraint contribute beyond teacher-score minimization alone? Table 8 shows that no candidates are infeasible at the reference learning rate. At the larger rate, does removing the constraint affect accuracy or stability, and how sensitive are the results to the budget δ=0.05\delta=0.05?
19 comments
AuthorsOpenReview ↗

Overview of Our Responses

Thank you for your constructive review. We appreciate your recognition of our motivation, theoretical analysis, and rollout-source intervention. We respond to your eight questions in order, addressing W1 under Q1, W2 under Q2–Q3, and W3 under Q4–Q8.

AuthorsOpenReview ↗

Response to Q1 & W1 (Part 1 / 3)

Q1 & W1: The additional advantage over simpler server-step adjustments is not sufficiently established, which limits the evidence for the proposed adaptive selector. How does FedTOPS compare with more thoroughly tuned fixed multipliers and FedExP?

A1: Thank you for your insightful comments. We added validation-tuned fixed scaling and validation-tuned FedExP, evaluated on all 66 benchmarks. FedTOPS achieves higher mean macro Avg@8\mathrm{Avg}@8 than both baselines in all 33 settings, while selecting its multiplier online within each training run.

Main Results:

R1-Table1 presents benchmark-level results for 0.6B/1×LR0.6\mathrm{B}/1\times\mathrm{LR}, and R1-Table2 summarizes all 33 settings.

R1-Table1: Performance against validation-tuned baselines across all 66 benchmarks under 0.6B/1×LR0.6\mathrm{B}/1\times\mathrm{LR}. Avg@8\mathrm{Avg}@8 and Pass@8\mathrm{Pass}@8 (%) are reported separately.

Avg@8\mathrm{Avg}@8

Method\mathrm{Method}MATH500\mathrm{MATH500}AMC23\mathrm{AMC23}Minerva\mathrm{Minerva}OlympiadBench\mathrm{OlympiadBench}AIME24\mathrm{AIME24}AIME25\mathrm{AIME25}
FedAvg\mathrm{FedAvg}11.39±0.4611.39 \pm 0.464.48±1.544.48 \pm 1.543.62±0.303.62 \pm 0.304.02±0.324.02 \pm 0.320.42±0.420.42 \pm 0.420.14±0.240.14 \pm 0.24
FedExP\mathrm{FedExP}11.60±0.6211.60 \pm 0.624.79±1.184.79 \pm 1.183.31±0.283.31 \pm 0.284.16±0.404.16 \pm 0.400.56±0.480.56 \pm 0.480.14±0.240.14 \pm 0.24
TunedFixed\mathrm{Tuned\\ Fixed}44.94‾±0.74\underline{44.94} \pm 0.7422.29‾±1.30\underline{22.29} \pm 1.3013.28‾±0.38\underline{13.28} \pm 0.3815.86±0.28\mathbf{15.86} \pm 0.281.39‾±0.64\underline{1.39} \pm 0.640.83±0.42\mathbf{0.83} \pm 0.42
FedTOPS\mathrm{FedTOPS}45.83±0.30\mathbf{45.83} \pm 0.3023.13±2.19\mathbf{23.13} \pm 2.1913.86±0.19\mathbf{13.86} \pm 0.1915.81‾±0.27\underline{15.81} \pm 0.271.81±1.05\mathbf{1.81} \pm 1.050.69‾±0.24\underline{0.69} \pm 0.24

Pass@8\mathrm{Pass}@8

Method\mathrm{Method}MATH500\mathrm{MATH500}AMC23\mathrm{AMC23}Minerva\mathrm{Minerva}OlympiadBench\mathrm{OlympiadBench}AIME24\mathrm{AIME24}AIME25\mathrm{AIME25}
FedAvg\mathrm{FedAvg}50.67±1.4750.67 \pm 1.4724.17±5.2024.17 \pm 5.2018.14±0.5618.14 \pm 0.5622.01±0.2322.01 \pm 0.233.33±3.333.33 \pm 3.331.11‾±1.92\underline{1.11} \pm 1.92
FedExP\mathrm{FedExP}51.13±1.2151.13 \pm 1.2130.00±7.5030.00 \pm 7.5017.16±1.4917.16 \pm 1.4921.71±1.2621.71 \pm 1.264.44‾±3.85\underline{4.44} \pm 3.851.11‾±1.92\underline{1.11} \pm 1.92
TunedFixed\mathrm{Tuned\\ Fixed}72.13‾±0.64\underline{72.13} \pm 0.6451.67‾±3.82\underline{51.67} \pm 3.8231.37‾±1.06\underline{31.37} \pm 1.0636.25‾±0.48\underline{36.25} \pm 0.487.78±5.09\mathbf{7.78} \pm 5.094.44±1.92\mathbf{4.44} \pm 1.92
FedTOPS\mathrm{FedTOPS}72.73±1.14\mathbf{72.73} \pm 1.1460.00±6.61\mathbf{60.00} \pm 6.6131.62±0.37\mathbf{31.62} \pm 0.3736.80±0.26\mathbf{36.80} \pm 0.267.78±1.92\mathbf{7.78} \pm 1.924.44±1.92\mathbf{4.44} \pm 1.92

Values are mean ± sample standard deviation over 33 training runs. Bold and underlined means indicate the highest and second-highest values in each column, including ties at the displayed precision. Following the original FedExP evaluation protocol, we evaluate the average of its last 22 global models.

AuthorsOpenReview ↗

Response to Q1 & W1 (Part 2 / 3)

R1-Table2: Summary across model sizes and learning rates. Six-benchmark macro Avg@8\mathrm{Avg}@8 and Pass@8\mathrm{Pass}@8 (%).

Method\mathrm{Method}0.6B/1×LR0.6\mathrm{B}/1\times\mathrm{LR} Avg@8\mathrm{Avg}@80.6B/1×LR0.6\mathrm{B}/1\times\mathrm{LR} Pass@8\mathrm{Pass}@80.6B/3×LR0.6\mathrm{B}/3\times\mathrm{LR} Avg@8\mathrm{Avg}@80.6B/3×LR0.6\mathrm{B}/3\times\mathrm{LR} Pass@8\mathrm{Pass}@81.7B/1×LR1.7\mathrm{B}/1\times\mathrm{LR} Avg@8\mathrm{Avg}@81.7B/1×LR1.7\mathrm{B}/1\times\mathrm{LR} Pass@8\mathrm{Pass}@8
FedAvg\mathrm{FedAvg}4.01±0.114.01 \pm 0.1119.90±0.6419.90 \pm 0.6410.18±2.4110.18 \pm 2.4129.97±1.4329.97 \pm 1.4311.88±0.2311.88 \pm 0.2337.94±1.7937.94 \pm 1.79
FedExP\mathrm{FedExP}4.09±0.074.09 \pm 0.0720.93±0.8120.93 \pm 0.8111.36†11.36^{\dagger}30.31†30.31^{\dagger}11.83†11.83^{\dagger}36.81†36.81^{\dagger}
TunedFixed\mathrm{Tuned\\ Fixed}16.43‾±0.27\underline{16.43} \pm 0.2733.94‾±1.00\underline{33.94} \pm 1.0017.31‾±0.36\underline{17.31} \pm 0.3636.26±1.66\mathbf{36.26} \pm 1.6625.61‾±0.44\underline{25.61} \pm 0.4446.05‾±0.45\underline{46.05} \pm 0.45
FedTOPS\mathrm{FedTOPS}16.85±0.33\mathbf{16.85} \pm 0.3335.56±1.16\mathbf{35.56} \pm 1.1617.51±0.17\mathbf{17.51} \pm 0.1735.09‾±1.14\underline{35.09} \pm 1.1426.14±0.15\mathbf{26.14} \pm 0.1547.41±0.66\mathbf{47.41} \pm 0.66

Values are mean ± sample standard deviation over 33 training runs; †^{\dagger} denotes a single run. Macro equally weights the 66 benchmarks.

Under 0.6B/1×LR0.6\mathrm{B}/1\times\mathrm{LR}, FedTOPS achieves higher mean Avg@8\mathrm{Avg}@8 than Tuned Fixed on 44 of the 66 benchmarks and matches or exceeds it in Pass@8\mathrm{Pass}@8 on all 66. Across the 33 settings, FedTOPS achieves the highest mean macro Avg@8\mathrm{Avg}@8, exceeding Tuned Fixed by 0.420.42, 0.200.20, and 0.530.53 percentage points, respectively. It also achieves higher macro Pass@8\mathrm{Pass}@8 in 22 of the 33 settings. These results demonstrate the effectiveness of adaptive scaling across model sizes and learning rates.

These performance gains come with low end-to-end runtime overhead. FedTOPS selects its multiplier online, with at most 13.213.2\\% additional runtime relative to FedAvg. Under our validation protocol, Tuned Fixed searches 66 full-budget candidates per setting and incurs 264264\\%–272272\\% overhead when the offline search cost is included, as detailed in Q2. FedTOPS therefore achieves higher mean macro Avg@8\mathrm{Avg}@8 across all 33 settings at substantially lower total cost.

Tuning protocol of two newly added baselines:

Both baselines were tuned on the same held-out 192192-question MATH validation set (6464 questions per client), disjoint from training and test data. Across the 33 settings, the search covered 1818 full-budget fixed-scaling configurations and 99 FedExP calibration runs, with 3636 validation evaluations in total. Existing fixed-scaling checkpoints were reused where available.

For Tuned Fixed, each multiplier in 0.5,1,2,3,5,10\\{0.5,1,2,3,5,10\\} was evaluated after 5050 training rounds. Round-5050 validation Avg@2\mathrm{Avg}@2 selected one multiplier per setting for the reported runs.

For FedExP, we searched 33 normalized values, c∈10−3,10−2,10−1c\in\\{10^{-3},10^{-2},10^{-1}\\}, with ϵ0=cSref\epsilon_0=cS_{\mathrm{ref}}. Each candidate ran for 1515 rounds using the 5050-round cosine schedule. Selection used mean validation Avg@2\mathrm{Avg}@2 at rounds 1010 and 1515, with a prespecified paired-bootstrap rule favoring larger ϵ\epsilon when validation differences were unresolved.

Here, SrefS_{\mathrm{ref}} is the first calibration round’s mean squared client-update norm. The selected ϵ0\epsilon_0 was fixed across formal runs, with ϵt=ϵ0(ηt/η0)2\epsilon_t=\epsilon_0(\eta_t/\eta_0)^2.

R1-Table3: Selected hyperparameters and FedExP extrapolation behavior.

Setting\mathrm{Setting}Fixed;a\mathrm{Fixed}\\;aFedExP;c\mathrm{FedExP}\\;cG/EG/Ea>1;roundsa>1\\;\mathrm{rounds}
0.6B/1×LR0.6\mathrm{B}/1\times\mathrm{LR}10100.10.10.9210.9210/1500/150
0.6B/3×LR0.6\mathrm{B}/3\times\mathrm{LR}220.010.010.8650.8650/500/50
1.7B/1×LR1.7\mathrm{B}/1\times\mathrm{LR}330.10.10.9090.9090/500/50

The last 22 columns summarize formal FedExP runs. EE is the mean squared client-update norm, and GG is the squared norm of their average. G/EG/E is averaged across communication rounds.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

For ease of reference, we refer to the reviewers as R1 (CuVx), R2 (pPY1), and R3 (XJxU) below.

We sincerely thank all reviewers for their thoughtful and constructive feedback. We uploaded the revised manuscript on November 15 (AoE). Below, we summarize our contributions and the main updates in the revised manuscript.


Core Contributions

  1. Understanding federated OPD. We provide the first systematic study of federated OPD and identify aggregation–rollout feedback: aggregation changes both the learner and the generator of future training data. This coupling helps explain why collaboration gains depend strongly on the local learning rate.

  2. Theoretical analysis and FedTOPS. We analyze how aggregation-induced optimization lag can delay access to useful teacher supervision, even under stable updates and a common optimum. Guided by this analysis, FedTOPS adapts the aggregate-update magnitude using cached teacher feedback and clientwise predictive-change constraints.

  3. Extensive evaluation with low overhead. Across six benchmarks and two student sizes, FedTOPS improves macro Avg@8 over FedAvg by 7.33–14.26 percentage points. It also achieves higher mean macro Avg@8 than Tuned Fixed with 69.5–74.5% lower total cost, including search.


Experiments Added in Response to Reviewer Feedback

  • Stronger baselines and multiple seeds (R1, R3): Tuned Fixed and FedExP comparisons; three-seed evaluation of FedAvg, fixed scaling, and FedTOPS across the three main settings. Section 4.2, Tables 1–3.
  • Larger learning rate (R3): three-seed comparisons at 10×LR. Section 4.4; Appendix I.2.
  • Expanded rollout-source experiments (R1, R3): rollout-source interventions at additional learner checkpoints and new fixed-source training controls. Section 4.3, Figures 3–4.
  • Deeper ablations (R1, R3): candidate-set choices, KL-budget sensitivity, and removal of the constraint. Section 4.4, Table 4; Appendices G.3 and G.6.
  • End-to-end efficiency (R1, R2, R3): accuracy versus total training and selection cost. Section 4.4, Figure 6.

Additional Clarifications

  • Practical motivation (R2, R3): a concrete financial-institution example illustrating the application of federated OPD. Section 1.
  • Relation to client drift (R2): discussion of how aggregation–rollout feedback relates to conventional client drift. Section 2.4; Appendices B.3–B.4.

The additional experiments used approximately 500 GPU-hours on NVIDIA RTX PRO 6000 GPUs. Our main claims, method, and core experimental design remain unchanged. All revisions are highlighted in blue. The extensive highlighting in the Experiments section mainly reflects the updated multi-seed results and the reorganization of tables and figures to make use of the additional page.

We believe these additions and clarifications address the reviewers’ main concerns. We sincerely appreciate the time and guidance of the area chair and reviewers and thank them for considering the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.