- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GUIDE-FBO is a Federated Bayesian Optimization framework where distributed agents optimize heterogeneous black-box tasks while keeping local data private. Agents share only compact distributions over local optima (low communication); the server merges/reweights these and sends global guidance back; each agent uses it only to rescale covariance while keeping its local posterior mean. It wins on synthetic benchmarks but gains negligibly on real-world tasks.
Critical strengths
1/Sound theory, few algebraic/logical errors. 2/Broad testing on12 synthetic + 3 real tasks with ablations. 3/Strong synthetic performance over competing FBO methods.
Critical weaknesses
1/ in the real-world tasks (Activity Recognition, FedHPO), FBO methods differ only marginally (accuracy 0.8511 vs 0.8513),gains are comparable, not superior, yet the method adds heavy overhead (DPGMM fitting, optimum extraction, server clustering) while cost not justified.
2/ in the "synthetic experiments", it shows that "federation fixed at N=16 agents", no scalability or computational-cost analysis; real federations may reach hundreds of agents.
3/ for "federated privacy", it shows that "agents upload Gaussian distribution components", the issue is that no theoretical privacy guarantee, shared components could leak an agent's objective or optimal design locations.
4/ "proofs only for GUIDE-UCB", the issue is that GUIDE-NEI and GUIDE-TS lack regret/convergence guarantees despite strong results.
5/ "GUIDE-TS on real tasks", it states "beats TS only on Landmine Detection", very weak real-world performance.
6/ "Sections 3.1 vs 3.2", it shows "m = posterior paths vs agent index", the issue is that symbol reuse conflict.
7/ in "Section 3.1 vs Appendix B.2", same symbol for scalar and matrix.
Questions for authors
see the above
2 comments
Response to Reviewer mwen (Part 2/2)
W4: Theoretical guarantees for GUIDE-TS and GUIDE-NEI
Our theoretical contribution is the regret and query-selection analysis of GUIDE-UCB, complemented by empirical evaluation with UCB, TS, and NEI. We do not claim convergence guarantees for all three instantiations.
Appendix B.3 establishes their common mathematical foundation: FI-GP preserves the local posterior mean and defines a positive-semidefinite covariance, hence a valid auxiliary GP. The regret argument in Lemma B.1 additionally uses the UCB selection inequality. TS and NEI use different selection rules and require separate convergence analyses.
Revisions: Opening of Section 4; Section 3.3 and Appendix B.3 for the common construction; Appendices B.4–B.5 for the UCB analysis.
W6–W7: Reused symbols
Thank you for catching these ambiguities. We now use for posterior-path samples, separating this index from the agent indices . We also use for the scalar number of active local mixture components, separating it from the notation used for kernel matrices.
Revisions: Sections 3.1–3.2 and Appendix I for sample indexing; Section 3.1 and Appendix B.2 for component-count and kernel-matrix notation.
W3: Privacy guarantees for the exchanged components
We agree that keeping raw observations local does not by itself provide differential privacy. The exchanged distributions can still reveal information about an agent's objective or promising design locations.
Our contribution is a communication-efficient FBO mechanism that exchanges compact distributions and uses them to guide local exploration while keeping raw observations on each agent. We do not claim that the current mechanism satisfies formal differential privacy.
We have considered four possible directions for a private extension:
- Perturb the posterior-optimum samples on each agent before fitting the mixture.
- Privatize intermediate statistics during mixture fitting.
- Privatize the released component parameters on each agent.
- Privatize aggregated components on a trusted server before redistribution.
Each route would require a guarantee for the chosen privacy unit, protection of the complete released information, and accounting across rounds. A key challenge for client-side privacy is preserving useful spatial guidance after adding noise calibrated to the sensitivity of the released information. Server-side aggregation may offer a better trade-off when individual contributions and aggregation weights are controlled, although the current clustering and weighting operations do not automatically provide a sensitivity reduction. This route would also require trust in the server.
These are directions for future work rather than privacy mechanisms already established in this paper. Section 6 explicitly states the absence of a formal privacy guarantee and identifies private distribution release and server aggregation as future directions.
Manuscript location: Section 3.1 defines the exchanged information; Section 6 states the privacy limitation and future directions.
Response to Reviewer mwen (Part 1/2)
Thank you for your time and careful assessment. We appreciate your positive evaluation of the theory and synthetic results, as well as your comments on missing experimental evidence and ambiguous notation. We have added scalability experiments and statistical comparisons, clarified the runtime and theoretical scope, and corrected the notation. Revisions are marked in blue in the manuscript.
W1 and W5: Real-world gains, GUIDE-TS, and computational cost
We added two-sided paired t-tests over 10 runs, pairing the mean performance across agents within each run. Holm correction covers 108 Level 2 synthetic comparisons and, separately, 27 original-budget real-world comparisons. GUIDE-UCB significantly outperforms UCB and CGP-UCB on 10 of 12 synthetic benchmarks each. Under the original real-world budget, neither comparison is significant; the only significant result among the 27 tests is GUIDE-TS versus FTS-DE on Landmine Detection. We acknowledge that these original-budget results provide limited evidence of a practical advantage, and have revised the manuscript to distinguish small numerical gains from statistically significant improvements.
These real-world tasks have only 2, 3, and 4 dimensions. With 80 evaluations per agent, final differences may be small, while 10 runs limit estimate precision. We therefore examined a smaller budget of 2d initial evaluations plus 5d BO rounds over 50 independent runs. Given the time available, this supplementary analysis focuses on Landmine Detection (d=2, 4+10 evaluations).
GUIDE-UCB achieves mean final AUC 0.786961 (SE 0.000418), versus 0.783679 for UCB and 0.785256 for CGP-UCB. Both gains are significant after Holm correction across nine matched paired t-tests (p=7.21×10−7 and 5.00×10−4, respectively). This provides evidence of GUIDE-UCB's sample efficiency on Landmine Detection under a limited evaluation budget.
Revisions: Section 5.2; Appendix C.5 (protocol); D.2, Tables 12–13 (synthetic); D.4, Table 17 (original real-world); D.5, Tables 18–19 and Figure 8 (low-budget Landmine).
Under the original budget, GUIDE-TS improves over independent TS only on Landmine Detection, with an average rank of 9.67 across the three tasks. A possible explanation is that TS maximizes a sampled path: guidance changes the path distribution, but a particular draw can still attain its maximum outside the guided regions. This may make the effect less consistent across rounds than with UCB or NEI.
The mean runtime comparison across 12 Level 2 synthetic benchmarks is:
GUIDE therefore requires more computation than independent BO but is faster than the corresponding CGP variants on average and on 9 of 12 benchmarks. These timings primarily measure optimizer overhead. In applications dominated by expensive objective evaluations, this overhead can be a small part of the total cost, making effective use of the evaluation budget the main consideration.
Revisions: Section 5.2 and Appendix D.4 for real-world results; Appendix D.1 discusses the TS sampling mechanism; Section 5.4 and Appendix H report runtime and its scope.
W2: Federation size and scalability
We added two studies on Level 2 Rastrigin, each with 10 runs and 30 initial evaluations plus 50 BO rounds per agent. The tables below report mean final regret; lower is better.
Varying the number of agents with d=10:
Varying dimension with N=8:
On Rastrigin, GUIDE-UCB achieves lower mean regret at every tested federation size and dimension, demonstrating a consistent advantage from 4 to 128 agents and from 2 to 40 dimensions. Increasing the number of agents does not monotonically improve performance, but the advantage over independent UCB is maintained throughout the tested range.
We also measured runtime as N increases. With clients executed sequentially, GUIDE-UCB takes 105.64 seconds at N=4 and 3107.76 seconds at N=128. After subtracting measured server time and dividing by N, the remaining time stays between 24.15 and 28.49 seconds. Server aggregation and packet generation take 0.080–16.199 seconds per run, less than 0.6% of total runtime. These measurements show that server processing contributes only a small fraction of runtime in this implementation.
The full results, SEs, and convergence plots are in Appendix F.1; the runtime breakdown is in Appendix H. Both studies are summarized in Section 5.4.