- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a framework to improve the training configuration of a federated model trained across several labs, with an LLM choosing a value for 4 hyperparameters such as the learning rate. The results of earlier trials are used as feedback.
Critical strengths
The paper sets out to address an important problem in the medical setting: distributed training where different laboratories can not share their data.
Critical weaknesses
The evaluation has flaws:
- no comparison to traditional hyperparameter tuning, not even to random search
- the laboratories are not real independent entities with a domain shift and batch effects, but random splits of a single dataset. The evaluation therefore never tests the hard part of combining multiple datasets across labs, but only addresses the distribution of the training.
- the authors' own pipeline, when training a single model ("direct"), performs worse than a fixed model
- similarly, other baselines also use 1 model, while BioCoLoop uses 10 labs
Questions for authors
The novelty of the work is not very clear to me, since the main approach appears to be asking an LLM to turn 4 knobs in the training hyperparameter space. Perhaps a better explanation of the novelty could make the reader appreciate the contribution better.
2 comments
I thank the authors for the revisions they made. I'm still not convinced that the novelty and demonstrated gains are at a high enough level, so I will not raise my score, but I wish them all the best with continuing this work.
Thank you for the review and for highlighting the setting we target: distributed training in biology and medicine when laboratories cannot share data. The revision adds single-laboratory random, Bayesian and exhaustive search, matched-access controls, a ten-laboratory fixed model, and laboratories built from distinct released studies. These separate two questions: whether collaborative training helps on real sources, and whether the research loop adds gain beyond that.
1. Traditional tuning and random search
On laboratory 0, random search, GP-EI Bayesian optimization and exhaustive enumeration all stay below collaborative BioCoLoop (Appendix D): DTI AUROC 90.10 vs 93.76, and random-search proteomics AP 20.13 vs 36.99.
At matched ten-laboratory access (Appendix E.1-E.2), all policies share laboratories, panels, configuration library and budget, and failed requests are charged. Language-model search and GP-EI reach 31.50 VCC Top-1 against 27.53 for random search. In eight-seed extensions registered before the new fits:
The research loop's extra gain over random search is small, and GP-EI performs better on protein (39.02 AP) and on the gene-disjoint cells (23.28 Top-1).
2. Simulated laboratories and domain shift: collaborative training on real sources
The main laboratories are grouped by canonical compounds (DTI), compound groups (proteomics) and disjoint training interventions (cells); none is shared between laboratories. The source-defined experiments assign a different released study to each laboratory (Section 4.5, Appendix B.2):
Not every source helps: BioSNAP lowers target DTI AUROC by 3.03, and Replogle and Nadig lower VCC Top-1 by 2.37 and 2.55, so compatibility between studies matters. These experiments measure the value of collaborative training, not the research loop's extra gain, which the matched-access controls above evaluate. All laboratories run on shared compute; real-site deployment is next.
3. Direct search versus the fixed model
Section 4.2 now defines both. Direct search trains several candidates without feedback from earlier evaluations and keeps one by development score; the fixed model is the single default configuration. The single-laboratory gap comes from selecting on laboratory 0's small development panel.
4. Ten laboratories versus one model
The ten laboratories hold data; they are not an ensemble. BioCoLoop, like every baseline, returns one predictor evaluated on the same test set. Table 1 makes the standard federated comparison, developing with collaborators against developing alone, and its caption now says so. It cannot separate the contribution of collaboration from that of search on its own, so two controls hold access fixed. Against a ten-laboratory fixed model (Table D.1), BioCoLoop adds 1.96 DTI AUROC and 6.77 Norman Top-1. On proteomics and VCC the fixed model already matches BioCoLoop (37.07 vs 36.99 AP; 31.54 Top-1 for both), and on Tahoe it is 3.48 points higher, so there the gain is carried by collaboration itself.
5. "Asking an LLM to turn 4 knobs"
The contribution is a protocol and interface through which laboratories that cannot pool data jointly decide which design to keep, what to try next and how to allocate further training, exchanging only model updates and bounded per-laboratory development evidence. It serves three task families and several research agents. The four-factor library keeps the comparisons interpretable, and Appendix E.6 widens it to 36 recipes.
The clearest gain from the loop itself is the two-stage budget allocation. At matched training and evaluation budgets, screening on early collaborative evidence and training the promising candidates longer raises Norman Top-1 from 11.48 to 22.32 over uniform allocation (+10.84, CI [4.54, 17.73]), and across 12 task-seed pairs it never loses to uniform (E.4). Random promotion after the same screen is competitive with trajectory promotion (six wins, six losses), so the gain comes mainly from screening followed by longer training, not from trajectory guidance alone.
Thank you again; these points led to the controls added in this revision.