- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes AssayRouter, a method for selecting frozen molecular predictors for a new assay with only a small number of labeled molecules. Instead of ranking source predictors by their standalone accuracy or similarity to the target, AssayRouter learns post-fit utility from completed historical assays that how much adding a source predictor improves the final locally fitted predictor. A shared utility regressor then ranks frozen sources for an unseen assay, selects the Top-4, and fits a convex combiner using the available target labels. The method is trained on 20 ChEMBL-MT pseudo-targets and evaluated on 24 external regression assays across six predictor interfaces and three label budgets (16/32/64).
Critical strengths
-
The paper introduces a new and practically relevant setting for molecular assay prediction. It formulates the problem of selecting a small subset of frozen predictors from previously completed assays to support a new assay with only a few labels, under the constraint that source training data and parameters are unavailable.
-
The post-fit utility formulation is motivated and specifically tailored to this setting. Rather than ranking source predictors by standalone quality or similarity, AssayRouter learns whether a source provides complementary value after being combined with the local target predictor.
-
The accompanying permutation, unseen-interface, and matched-control experiments provide relatively strong evidence that this transferred utility signal is meaningful.
Critical weaknesses
-
The empirical scope is relatively narrow. The current evidence is restricted to three scarce-label budgets (16/32/64) and homogeneous interface banks. It is therefore unclear how broadly the conclusions extend to other label regimes or heterogeneous predictor banks.
-
The deployment contract is only weakly tested. The fixed action was informed by earlier sourcewise studies on ExpansionRx, Biogen, and TDC, which are also used for the main external evaluation. Although the evaluation targets are excluded from router fitting, it remains unclear whether the advantage persists under alternative fan-in values, combiners, or support allocations.
-
The learned historical prior is based on only 20 pseudo-target assays. The paper does not sufficiently characterize how performance depends on the size, diversity, or representativeness of the historical assay panel, which is important for assessing transfer to substantially different assay banks.
-
The improvement over the strongest matched baseline is modest and concentrated in the lowest-label regime. Against Support-CV@4, the strict NLL gain decreases from 0.0736 at 16 labels to 0.0413 at 32 and 0.0078 at 64 labels, while the 64-label and Biogen confidence intervals cross zero. This suggests that the practical benefit may be limited mainly to highly data-scarce settings.
-
The presentation could be improved, e.g. the text and annotations in Figure 1 are quite small and difficult to read at normal page scale.
Questions for authors
-
How sensitive are the main results to the fixed fan-in ? Since this choice was informed by retrospective studies on ExpansionRx, Biogen, and TDC, which are also used for final evaluation, could the authors clarify the development/evaluation boundary and report results for alternative values such as and ?
-
The advantage over Support-CV@4 becomes very small at 64 labels. What range of target-data scarcity do the authors consider the intended operating regime of AssayRouter, and do results at larger label budgets continue this trend?
-
How sensitive is AssayRouter to the number and diversity of historical pseudo-target assays? For example, does performance improve consistently as more historical assays are included, or is the learned utility prior dependent on particular assay families?
-
The current method fixes the number of selected predictors to four for every target. It would be interesting to explore target-adaptive fan-in, allowing AssayRouter to select different numbers of predictors depending on the target assay and available evidence, which may better balance complementary information against overfitting or source interference.
3 comments
Thank you for the detailed response and the additional experiments. These new results partially address several of my previous concerns.
However, some concerns remain. The additional experiments clarify that AssayRouter is primarily effective in the highly data-scarce regime, which the authors explicitly acknowledge as the intended operating setting. However, this also narrows the practical scope of the method. The results also indicate sensitivity to the choice of combiner, while the evaluation remains limited to homogeneous predictor banks.
I also note that several important experiments and clarifications presented in the rebuttal have not been incorporated into the revised manuscript. The presentation issue I previously mentioned also remains: Figure 1 is still difficult to read.
More generally, I believe the manuscript still requires substantial polishing before publication. Several texts are difficult to follow, and the overall presentation is not yet sufficiently clear or concise. Reviewer Xctr also assigned a score of 1 for Presentation, which is consistent with this concern. I also noticed multiple awkward line breaks, including lines containing only one or two words and unnecessary blank spaces, which further affect readability and make the manuscript appear insufficiently polished.
Therefore, I will maintain my current score.
Thank you for engaging with the experiments and for the specific feedback. We address each remaining point below.
Operating range as a design target. The 16–32 label regime is the typical condition in prospective assay campaigns, where compounds are expensive and target labels are scarce. At 64–128 labels, simpler selectors already perform well, and we report this explicitly. The method is designed for the setting where selection matters most, and we view the diminishing advantage at larger budgets as expected rather than limiting.
Combiner sensitivity. The router's training target measures improvement after the submitted Ridge/NNLS fit. The TabPFN combiner result confirms that the learned scores are specific to this fitting procedure. We see this as a clear design coupling, not an uncharacterized fragility, and have stated the scope in our response. A practitioner changing the combiner would retrain the router with matching labels.
Homogeneous banks. Table 20 trains the router on five predictor families and evaluates on the sixth, with positive transfer in all six cases. This tests cross-family generalization within our evaluation. We agree that mixed-family banks in a single deployment remain untested and have marked this boundary.
Presentation. We take this concern seriously. The revision plan includes: (1) incorporating all six supplementary experiments and their tables into the appendix, (2) redrawing Figure 1 at higher resolution with readable annotations, (3) fixing the line-break and spacing artifacts throughout the manuscript, and (4) tightening the main-text exposition following both your and Reviewer Xctr's presentation feedback. The current system does not permit uploading a revised manuscript during the discussion period, but we commit to completing these changes for the camera-ready version and are prepared to share a draft with the reviewers or Area Chair upon request.
We believe the scientific contribution, the supervision principle and its controlled validation, is now supported by the experiments in the rebuttal. We hope the planned presentation improvements address the remaining barrier, and welcome any further suggestions.
Source budget, historical panels, label range, combiner and allocation
We report new experiments on source budget, historical panel size, label range, combiner and support allocation. The strict advantage holds at K=2 and K=4. It narrows at K=8 and at 128 labels, and reverses when the combiner is replaced by TabPFN. All five prespecified ten-assay historical panels retain a strict advantage over Support-CV.
Source budget and development data
We apply the same scoring rule at each source budget. With the trained router fixed, the strict advantage persists at K=2 and K=4. Earlier studies on Biogen, ExpansionRx and TDC informed the choice of four sources (Appendix C). All 24 external assays were held out of router fitting and capacity selection. The three-way comparison uses 18 ExpansionRx and TDC targets, since Biogen has five sources.
At K=2, the gain across all 24 targets is 0.0360 [0.0106, 0.0622]. At K=8, ExpansionRx uses its entire bank, so both selectors make identical predictions. On TDC we additionally widen Support-CV's screen to 12, giving 495 eight-source subsets. C retains a strict advantage of 0.0306 [0.0095, 0.0533] there. The prediction-averaged NLLs are 1.9121 for C and 1.9008 for Support-CV.
Each bank uses one interface family. Table 20 tests a family held out of historical training. A target-adaptive K would require a rule for deciding when the next source is worth its cost. Here K is supplied at deployment.
Historical panel size and composition
The five prespecified ten-assay panels all outperform Support-CV in strict NLL. At 5, 10 and 15 historical assays, the five-panel mean NLLs are 2.1318, 2.1148 and 2.1142, compared with 2.1102 for the complete panel of 20. Composition matters most at the smallest size. Five-assay panels range from 2.1137 to 2.1836, while ten-assay panels range from 2.1122 to 2.1176. The mean gains over Support-CV with 10 and 15 assays are 0.0364 [0.0185, 0.0557] and 0.0369 [0.0186, 0.0565], respectively.
Removing each of ten endpoint groups in turn retains a strict advantage of 0.0323–0.0396 over Support-CV. We kept the source bank fixed and used five size-matched random deletions for each deletion size. Removing solubility assays adds 0.0040 NLL [0.0003, 0.0087] relative to these controls. These interventions measure the effect of which ChEMBL-MT assays supply supervision.
Target labels, combiner and support allocation
The clearest benefit is at 16 and 32 labels, where the submitted strict NLL gains are 0.0736 and 0.0413 (Table 23). At 64 labels the gain is 0.0078 and its interval crosses zero, as does the Biogen interval. At 128 labels we used the same confirmation molecules, all 24 targets and six interfaces. The frozen C router scores 1.9158 strict NLL versus 1.9060 for Support-CV. The paired difference (Support-CV minus C) is −0.0097 [−0.0314, 0.0107]. Averaging favors Support-CV, 1.8561 versus 1.8778. The advantage is clearest in the low-label regime for which the router was designed.
The choice of combiner matters. On identical fitting labels and with the original four-source selections frozen, TabPFN gives strict NLLs of 2.3312 for C's sources and 2.3111 for Support-CV's sources. The difference favors Support-CV by 0.0202 [0.0051, 0.0377]. After averaging, the NLLs are 2.2994 and 2.2764. The historical utility target measures improvement after the submitted Ridge/NNLS fit, and that is the procedure for which the selected sources improve strict NLL.
Varying only the allocation of the same support molecules between routing (R) and fitting (C) preserves the strict advantage at all three ratios below.
All three selectors, including Support-greedy, reach their lowest strict NLL with three quarters of the labels allocated to fitting. Under prediction averaging, the gains for C are −0.0001, 0.0056 and 0.0289 at the respective ratios. Only the interval for 3:1 lies above zero.