acceptodds
Under review as a conference paper at ICLR 2027

AssayRouter: Historical Utility Priors for Frozen Molecular Predictor Routing

Abstract

Laboratories often face a new molecular assay with 16–64 labels and a bank of predictors whose training data and parameters are unavailable. The practical question is which frozen outputs to include in a small local model. AssayRouter treats completed assays as pseudo-targets and labels each candidate by its post-fit utility: the reduction in held-out discovery loss when the candidate is added to the local target predictor. A shared regressor learns to predict this utility from candidate behavior on the support set, without source identity; on a new assay, one frozen ranking selects four sources and separate labels fit a convex combiner. We train only on completed ChEMBL-MT assays and evaluate 24 external regression assays across six frozen interface families. AssayRouter-C lowers strict four-call negative log-likelihood (NLL) by 0.0409 relative to Support-CV@4. Frozen candidate-label permutations confirm that candidate–utility correspondence carries the transferred information, and leave-one-interface-out training shows that the mapping generalizes to unseen predictor families. Completed assays therefore provide transferable supervision for scarce-label routing through frozen prediction interfaces.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Molecular property prediction, Transfer learning, Meta-learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_GbxEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AssayRouter, a method for selecting frozen molecular predictors for a new assay with only a small number of labeled molecules. Instead of ranking source predictors by their standalone accuracy or similarity to the target, AssayRouter learns post-fit utility from completed historical assays that how much adding a source predictor improves the final locally fitted predictor. A shared utility regressor then ranks frozen sources for an unseen assay, selects the Top-4, and fits a convex combiner using the available target labels. The method is trained on 20 ChEMBL-MT pseudo-targets and evaluated on 24 external regression assays across six predictor interfaces and three label budgets (16/32/64).

Critical strengths
  1. The paper introduces a new and practically relevant setting for molecular assay prediction. It formulates the problem of selecting a small subset of frozen predictors from previously completed assays to support a new assay with only a few labels, under the constraint that source training data and parameters are unavailable.

  2. The post-fit utility formulation is motivated and specifically tailored to this setting. Rather than ranking source predictors by standalone quality or similarity, AssayRouter learns whether a source provides complementary value after being combined with the local target predictor.

  3. The accompanying permutation, unseen-interface, and matched-control experiments provide relatively strong evidence that this transferred utility signal is meaningful.

Critical weaknesses
  1. The empirical scope is relatively narrow. The current evidence is restricted to three scarce-label budgets (16/32/64) and homogeneous interface banks. It is therefore unclear how broadly the conclusions extend to other label regimes or heterogeneous predictor banks.

  2. The deployment contract is only weakly tested. The fixed K=4K=4 action was informed by earlier sourcewise studies on ExpansionRx, Biogen, and TDC, which are also used for the main external evaluation. Although the evaluation targets are excluded from router fitting, it remains unclear whether the advantage persists under alternative fan-in values, combiners, or support allocations.

  3. The learned historical prior is based on only 20 pseudo-target assays. The paper does not sufficiently characterize how performance depends on the size, diversity, or representativeness of the historical assay panel, which is important for assessing transfer to substantially different assay banks.

  4. The improvement over the strongest matched baseline is modest and concentrated in the lowest-label regime. Against Support-CV@4, the strict NLL gain decreases from 0.0736 at 16 labels to 0.0413 at 32 and 0.0078 at 64 labels, while the 64-label and Biogen confidence intervals cross zero. This suggests that the practical benefit may be limited mainly to highly data-scarce settings.

  5. The presentation could be improved, e.g. the text and annotations in Figure 1 are quite small and difficult to read at normal page scale.

Questions for authors
  1. How sensitive are the main results to the fixed fan-in K=4K=4? Since this choice was informed by retrospective studies on ExpansionRx, Biogen, and TDC, which are also used for final evaluation, could the authors clarify the development/evaluation boundary and report results for alternative values such as K=2K=2 and K=8K=8?

  2. The advantage over Support-CV@4 becomes very small at 64 labels. What range of target-data scarcity do the authors consider the intended operating regime of AssayRouter, and do results at larger label budgets continue this trend?

  3. How sensitive is AssayRouter to the number and diversity of historical pseudo-target assays? For example, does performance improve consistently as more historical assays are included, or is the learned utility prior dependent on particular assay families?

  4. The current method fixes the number of selected predictors to four for every target. It would be interesting to explore target-adaptive fan-in, allowing AssayRouter to select different numbers of predictors depending on the target assay and available evidence, which may better balance complementary information against overfitting or source interference.

3 comments
AuthorsOpenReview ↗

Source budget, historical panels, label range, combiner and allocation

We report new experiments on source budget, historical panel size, label range, combiner and support allocation. The strict advantage holds at K=2 and K=4. It narrows at K=8 and at 128 labels, and reverses when the combiner is replaced by TabPFN. All five prespecified ten-assay historical panels retain a strict advantage over Support-CV.

Source budget and development data

We apply the same scoring rule at each source budget. With the trained router fixed, the strict advantage persists at K=2 and K=4. Earlier studies on Biogen, ExpansionRx and TDC informed the choice of four sources (Appendix C). All 24 external assays were held out of router fitting and capacity selection. The three-way comparison uses 18 ExpansionRx and TDC targets, since Biogen has five sources.

SourcesAssayRouter-C strict NLLSupport-CV strict NLLSupport-CV minus C (paired 95% interval)
22.26242.30860.0462 [0.0143, 0.0788]
42.22382.27810.0543 [0.0307, 0.0792]
82.23262.24130.0087 [−0.0005, 0.0194]

At K=2, the gain across all 24 targets is 0.0360 [0.0106, 0.0622]. At K=8, ExpansionRx uses its entire bank, so both selectors make identical predictions. On TDC we additionally widen Support-CV's screen to 12, giving 495 eight-source subsets. C retains a strict advantage of 0.0306 [0.0095, 0.0533] there. The prediction-averaged NLLs are 1.9121 for C and 1.9008 for Support-CV.

Each bank uses one interface family. Table 20 tests a family held out of historical training. A target-adaptive K would require a rule for deciding when the next source is worth its cost. Here K is supplied at deployment.

Historical panel size and composition

The five prespecified ten-assay panels all outperform Support-CV in strict NLL. At 5, 10 and 15 historical assays, the five-panel mean NLLs are 2.1318, 2.1148 and 2.1142, compared with 2.1102 for the complete panel of 20. Composition matters most at the smallest size. Five-assay panels range from 2.1137 to 2.1836, while ten-assay panels range from 2.1122 to 2.1176. The mean gains over Support-CV with 10 and 15 assays are 0.0364 [0.0185, 0.0557] and 0.0369 [0.0186, 0.0565], respectively.

Removing each of ten endpoint groups in turn retains a strict advantage of 0.0323–0.0396 over Support-CV. We kept the source bank fixed and used five size-matched random deletions for each deletion size. Removing solubility assays adds 0.0040 NLL [0.0003, 0.0087] relative to these controls. These interventions measure the effect of which ChEMBL-MT assays supply supervision.

Target labels, combiner and support allocation

The clearest benefit is at 16 and 32 labels, where the submitted strict NLL gains are 0.0736 and 0.0413 (Table 23). At 64 labels the gain is 0.0078 and its interval crosses zero, as does the Biogen interval. At 128 labels we used the same confirmation molecules, all 24 targets and six interfaces. The frozen C router scores 1.9158 strict NLL versus 1.9060 for Support-CV. The paired difference (Support-CV minus C) is −0.0097 [−0.0314, 0.0107]. Averaging favors Support-CV, 1.8561 versus 1.8778. The advantage is clearest in the low-label regime for which the router was designed.

The choice of combiner matters. On identical fitting labels and with the original four-source selections frozen, TabPFN gives strict NLLs of 2.3312 for C's sources and 2.3111 for Support-CV's sources. The difference favors Support-CV by 0.0202 [0.0051, 0.0377]. After averaging, the NLLs are 2.2994 and 2.2764. The historical utility target measures improvement after the submitted Ridge/NNLS fit, and that is the procedure for which the selected sources improve strict NLL.

Varying only the allocation of the same support molecules between routing (R) and fitting (C) preserves the strict advantage at all three ratios below.

R:CAssayRouter-CSupport-CVStrict NLL gain (paired 95% interval)
1:32.08632.11270.0264 [0.0094, 0.0463]
1:12.11022.15110.0409 [0.0209, 0.0619]
3:12.18602.25360.0675 [0.0378, 0.0990]

All three selectors, including Support-greedy, reach their lowest strict NLL with three quarters of the labels allocated to fitting. Under prediction averaging, the gains for C are −0.0001, 0.0056 and 0.0289 at the respective ratios. Only the interval for 3:1 lies above zero.

Thank you for the detailed response and the additional experiments. These new results partially address several of my previous concerns.

However, some concerns remain. The additional experiments clarify that AssayRouter is primarily effective in the highly data-scarce regime, which the authors explicitly acknowledge as the intended operating setting. However, this also narrows the practical scope of the method. The results also indicate sensitivity to the choice of combiner, while the evaluation remains limited to homogeneous predictor banks.

I also note that several important experiments and clarifications presented in the rebuttal have not been incorporated into the revised manuscript. The presentation issue I previously mentioned also remains: Figure 1 is still difficult to read.

More generally, I believe the manuscript still requires substantial polishing before publication. Several texts are difficult to follow, and the overall presentation is not yet sufficiently clear or concise. Reviewer Xctr also assigned a score of 1 for Presentation, which is consistent with this concern. I also noticed multiple awkward line breaks, including lines containing only one or two words and unnecessary blank spaces, which further affect readability and make the manuscript appear insufficiently polished.

Therefore, I will maintain my current score.

Thank you for engaging with the experiments and for the specific feedback. We address each remaining point below.

Operating range as a design target. The 16–32 label regime is the typical condition in prospective assay campaigns, where compounds are expensive and target labels are scarce. At 64–128 labels, simpler selectors already perform well, and we report this explicitly. The method is designed for the setting where selection matters most, and we view the diminishing advantage at larger budgets as expected rather than limiting.

Combiner sensitivity. The router's training target measures improvement after the submitted Ridge/NNLS fit. The TabPFN combiner result confirms that the learned scores are specific to this fitting procedure. We see this as a clear design coupling, not an uncharacterized fragility, and have stated the scope in our response. A practitioner changing the combiner would retrain the router with matching labels.

Homogeneous banks. Table 20 trains the router on five predictor families and evaluates on the sixth, with positive transfer in all six cases. This tests cross-family generalization within our evaluation. We agree that mixed-family banks in a single deployment remain untested and have marked this boundary.

Presentation. We take this concern seriously. The revision plan includes: (1) incorporating all six supplementary experiments and their tables into the appendix, (2) redrawing Figure 1 at higher resolution with readable annotations, (3) fixing the line-break and spacing artifacts throughout the manuscript, and (4) tightening the main-text exposition following both your and Reviewer Xctr's presentation feedback. The current system does not permit uploading a revised manuscript during the discussion period, but we commit to completing these changes for the camera-ready version and are prepared to share a draft with the reviewers or Area Chair upon request.

We believe the scientific contribution, the supervision principle and its controlled validation, is now supported by the experiments in the rebuttal. We hope the planned presentation improvements address the remaining barrier, and welcome any further suggestions.

Reviewer_XctrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper addresses a few-shot prediction problem in drug discovery, as a new assay has only 16–64 labeled molecules, and a bank of previously trained models for other assays is available only as black boxes. The authors combine a simple local model, trained on the few labels, with four selected models from the bank as a learned weighted average. Their contribution, AssayRouter, is the selection step. On 20 completed historical assays, they measure each old model’s “post-fit utility,” the reduction in error when it is added to the local model, and train a gradient-boosting regressor to predict this utility from simple statistics of the model’s predictions on the few labels. On a new assay, the regressor scores all candidates once, and the top four are combined, avoiding a search over subsets. Evaluated on 24 external assays across six model families and three label budgets, the method modestly outperforms a cross-validated subset-search baseline, mainly at the smallest label budget, and controlled tests (label permutations, held-out model families) indicate that the learned relationship between model statistics and utility transfers to new assays.

Critical strengths
  1. Rigorous and well-controlled experimental design

The authors exclude all test assays from router training and model selection, keep the routing, fitting, and test data strictly separate, and compare against baselines that share the same model bank, four-model budget, and final combiner. The authors provide thorough ablation studies that convincingly isolate the contribution of each component of their method.

  1. Supervision aligned with the deployment decision

Training the selector on each model’s added value after combination with the local model, rather than on standalone accuracy, directly targets the decision being made. The clear gap to the residual-correlation baseline supports this choice.

Critical weaknesses
  1. Clarity.

The paper is written in an extremely dense, jargon-heavy style that hides a conceptually simple method. Key variants are defined only briefly (e.g. the Z variant appears only in a table caption and the appendix), which makes the paper hard to follow and limits its accessibility. This also affects the results section, which reads like a list of separate tests and numbers. A more connected narrative, explaining what each ablation tests and how the results build on each other, would make the findings much easier to follow

  1. Limited demonstrated benefit.

The improvement over Support-CV@4 is small (0.041 NLL under strict scoring, Table 1) and vanishes under cross-fit scoring (0.003, reversing at 64 labels, Table 22). The efficiency argument rests on counts of avoided NNLS fits, each a very small problem, without any wall-clock comparison. The practical significance of the method therefore remains unclear.

Questions for authors
  1. Can you report wall-clock times for AssayRouter and Support-CV, including feature construction and router inference, to substantiate the efficiency claim?

  2. Both AssayRouter and Support-CV achieve clearly lower NLL under cross-fit scoring (around 2.04, Table 22) than under strict scoring (around 2.11, Table 1), so a practitioner would likely prefer the averaged ensemble. In that setting, however, the advantage over Support-CV largely disappears (0.003 NLL, Table 22). Why is strict scoring with a fixed budget of four model calls the appropriate deployment scenario, and in what practical setting would this budget be binding?

3 comments
AuthorsOpenReview ↗

Strict and averaged scoring, wall-clock cost and variant definitions

We clarify the variant definitions and provide the numbers you requested. Under strict scoring, AssayRouter-C gains 0.0409 NLL over Support-CV with four sources. After prediction averaging, it gains 0.0056 with an interval spanning zero but uses 34.4% fewer distinct sources. The complete predictor builds in a median of 0.3085 s versus 0.6164 s.

Four-source and averaged prediction

We use strict scoring to ask how well one four-source choice predicts a new molecule. Averaging 16 fitted predictors measures what can be achieved with access to the union of their sources. We compare the two scoring rules with each variant held fixed. Tables 1 and 23 give the strict result for the centered C model. Table 22 gives the averaged result for the uncentered Δ model (Appendix A.2.1, lines 1071–1073). Positive differences below mean lower NLL than Support-CV.

AssayRouter variantStrict four-source NLL gainNLL gain after averaging 16 predictions
Δ (submitted Table 14)0.03690.0030
C (Table 23 and new paired replay)0.04090.0056

For C, the paired 95% intervals are [0.0209, 0.0619] under strict scoring and [−0.0139, 0.0259] after averaging. With 64 labels, the averaged difference is −0.0155 in favor of Support-CV. Averaging improves absolute NLL for both methods (2.0392 for C and 2.0448 for Support-CV overall). It also changes what must be available for each new molecule. One constituent needs four source outputs. The 16-constituent averages use a mean of 6.85 distinct sources for C and 10.45 for Support-CV, a 34.4% reduction.

Four was an empirical choice during earlier development on Biogen, ExpansionRx and TDC (Appendix C). One setting where a cap would bind is a screening campaign with a per-compound allowance for separately billed predictor calls. Strict scoring gives the accuracy attainable at that allowance, while routing on the labeled support R still uses the full bank. The 24 evaluation assays were held out of router fitting and capacity selection. On the same 18 targets admitting K=2, 4 and 8, the strict NLL gains over Support-CV were 0.0462, 0.0543 and 0.0087. The paired intervals exclude zero at two and four sources and cross zero at eight.

Measured cost

Including feature construction and router inference, the complete predictor took a median of 0.3085 s for AssayRouter-C and 0.6164 s for Support-CV. Preparing the routing inputs, including the learned scores for C, took a median of 155.92 ms for C versus 61.91 ms for Support-CV. Selection from those inputs took 0.20 ms versus 389.03 ms. We timed 432 target–interface–budget combinations on one Intel Xeon Gold 6338 CPU thread, with three repetitions per method and rotated order.

Complete predictor, median secondsBiogenExpansionRxTDC
AssayRouter-C0.25730.37230.3108
Support-CV0.21720.69200.6229
Support-greedy0.26230.45110.5449

Biogen's five-source bank makes subset search cheap enough that Support-CV is faster there. The timer covers routing inputs, features, scoring, source access, selection, final fitting and confirmation prediction with reuse across directions. Models and molecular representations are loaded beforehand. Four families in Biogen and ExpansionRx execute fitted heads. Their two neural families and all TDC interfaces read precomputed outputs. An independently timed single direction, with warmed routing-source access, takes 25.49 ms for C and 43.50 ms for Support-CV. Dividing the 782.1 s historical cost by the mean saving on this workload gives an estimated break-even of 2,529 complete predictor evaluations.

Reading the variants

The variants use the same 11 candidate measurements and historical refits. Each candidate is added to the local predictor, and held-out discovery molecules measure its contribution. Δ retains the raw loss reduction, C centers it within the historical block, and Z also divides by the block standard deviation (Table 1 and Appendix A.1.5). Section 4.2, lines 318–320, compares these variants. The permutation experiment in §4.1 tests whether candidate-specific utility matters. Holding out a predictor family tests whether the relationship transfers. The four-source comparison in §4.2 tests its value on a new assay.

We thank the authors for the detailed response and are raising our score to 3.

The wall-clock measurements fully address our first question, and the additional K sweep over 2, 4 and 8 is useful evidence that the advantage is budget-dependent. We appreciate that the authors state plainly where the paired intervals cross zero and where the averaged result reverses at 64 labels. We ask that all of the material provided in the response, including the timing results, the K sweep, the distinct-source counts and the stated limits of the comparison, be incorporated into the paper itself rather than left in the rebuttal.

Our clarity concern remains. The response explains the Δ, C and Z variants largely by pointing to where they are already defined. We would like a commitment to move those definitions into the main text and to restructure Section 4.2 so each ablation is introduced by what it tests. The paper should also state why C rather than Δ is the reported variant, and note that the reported timings exclude model loading.

AuthorsOpenReview ↗

Thank You and Commitment to Revisions

We sincerely thank the reviewer for the thoughtful follow-up and for raising the score.

We will incorporate all of the material from the rebuttal into the paper itself, including the wall-clock timing results, the K sweep over 2, 4 and 8 (with K = 4 being the default setting used throughout the main experiments), the distinct-source counts, and the stated limits of the comparison, including the cases where the paired intervals cross zero and the reversal at 64 labels.

We also agree that the clarity issues should be fixed in the paper. In the final version, we will:

move the definitions of the Δ, C and Z variants into the main text; restructure Section 4.2 so that each ablation is introduced by what it tests; explicitly state why C rather than Δ is the reported variant; note that the reported timings exclude model loading.

We thank the reviewer again for the constructive feedback, which has helped improve the paper.

Reviewer_xkfYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AssayRouter addresses molecular assays with fewer labels, when training a reliable model is challenging. It uses frozen predictors for a new assay prediction, where each candidate predictor is represented by 11 features. The post-fit utility of AssayRouter measure whether adding a source predictor actually improves the local model prediction.

Critical strengths

1- AssayRouter introduces a post-fit utility method where frozen models are selected based on their contribution to the actual improvement of the local model. 2- AssayRouter is evaluated on 24 external assays across three datasets and six predictors families, with several ablations to support the generalization.

Critical weaknesses

1- The method looks relatively hand-engineered to me, only for the training dataset at hand. While the same 11 candidate features can be computed for a new data, it would be interesting to understand (i) how robust these candidate features will be for a new train data, and (ii) how much additional hand-engineering is required to make them appropriate for AssayRouter, if we use some other data than ChEMBL-MT as training set.

2- There are too many mathematical notations and equations in the draft which makes the AssayRouter difficult to follow and have to move back and forth. I recommend not to present every simple technique with an equation/notation for example equ. 14 of selecting top-K.

Questions for authors

The paper mentions “The 11 coordinates separate complementarity from reliability.”, at line 143, how did the AssayRouter decides these 11 candidate features is the best choice? Is AssayRouter ablated with a different subset or number of features?

2 comments
AuthorsOpenReview ↗

Feature ablation, transfer and historical sensitivity

We respond with new feature-group ablations and cross-family transfer experiments. Removing the residual-alignment or output-distribution group raises strict NLL by 0.0322 or 0.0460. The uncertainty and metadata groups have intervals crossing zero. The features transfer to held-out predictor families without retraining.

Why we chose these 11 measurements

Each source gets eleven observations on the current assay. We want to know whether it can correct errors the local predictor makes. Two sources with the same error correlation but different output amplitudes contribute differently to a convex fit. The regressor receives both, along with member disagreement and sample sizes.

For each routing molecule, let v be the source prediction, r the observed target response minus the out-of-fold local Ridge prediction on the routing scale, and u the standard deviation of predictions across that source's frozen members. The summaries below are taken across routing molecules (Eq. 8 and Appendix A.1.7). We retrained after each group deletion with the same utility labels and historical assay-grouped capacity search. The full model scores 2.1102 strict NLL.

CoordinatesMeasurements available when selecting a sourceStrict NLL after deletionChange (paired 95% interval)
1–3. Alignment with local errorsMean of v × r, absolute value of that mean, Pearson correlation of v and r2.1425+0.0322 [0.0079, 0.0671]
4–6. Source output distributionMean of v, standard deviation of v, mean of the absolute values of v2.1563+0.0460 [0.0197, 0.0795]
7–9. Member disagreement and signal-to-noiseMean of u, standard deviation of u, SD(v) / max(mean(u), 0.001)2.1130+0.0028 [−0.00004, 0.0059]
10–11. Amount of dataln(1 + declared source training count) / 12 in the implementation, and log₂(number of routing labels)2.1098−0.0005 [−0.0065, 0.0049]

The first two coordinates expose direction and magnitude of the same alignment. Coordinate 9 compares variation across molecules with typical member disagreement on a molecule. The routing label count gives the sample size behind each alignment estimate. Six interface indicators enter separately. The regressor learns from historical refits how these quantities jointly predict added value. Ranking by residual correlation alone gives strict NLL 2.2947 under the same four-source budget and final fit (Table 1 and Appendix A.2.2).

We would retain the residual and output groups on the evidence of these refits. Deleting them one at a time also raises prediction-averaged NLL by 0.0226 and 0.0195, respectively. The strict interval for the disagreement group spans zero. For an interface that omits training size, the nine-coordinate refit dropping both size measures is an evaluated option. It scored 2.1098 versus 2.1102 with all eleven. Each deletion measures a group's contribution with the other groups present.

Table 20 applies the same measurements to a predictor family held out of historical training. Each router trains on the other five families. Permuting the candidate–utility correspondence worsens pooled NLL by 0.0497 [0.0135, 0.0963], with positive point differences in all six held-out families.

Changing the historical assays

All five prespecified ten-assay panels retained a strict NLL advantage over Support-CV. Across five nested sampling sequences, the mean NLLs for 5, 10 and 15 historical assays were 2.1318, 2.1148 and 2.1142. The full 20-assay model scored 2.1102. The range across panels narrowed from 2.1137–2.1836 with five assays to 2.1122–2.1176 with ten.

We also removed each of ten endpoint groups in turn while keeping the frozen source bank fixed. All ten resulting routers retained a strict advantage of 0.0323–0.0396 over Support-CV. Removing solubility assays added 0.0040 loss relative to five random deletions of the same size. These experiments test changes to historical supervision within ChEMBL-MT. For a different database, one recomputes features and loss reductions on its historical assays and refits the regressor with assay-grouped validation.

Data roles and notation

During historical training, R provides the routing features. Separate labels C fit the local-only and candidate-augmented predictors. Discovery molecules measure their loss difference. On a new assay, R fixes the four sources and C fits their weights. Equation 12 specifies the selection rule. Equation 14 shows why shifting or positively rescaling scores preserves that choice.

Reviewer_xkfYOpenReview ↗

Review update

Thank you to the authors for incorporating the updates and I hope you also feel it is an even better manuscript now.

I raise my score for the updated manuscript to a 3.

Reviewer_X95HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AssayRouter, a meta-learning approach for selecting frozen molecular predictors for new assays with limited labels. It trains a meta-model to estimate which frozen predictor outputs can improve a local Ridge regression model. The method uses engineered features describing predictor behavior and a histogram gradient boosting regressor to predict their utility. Experiments demonstrate modest improvements over support-based cross-validation while reducing inference-time model selection costs.

Critical strengths

The problem setting is interesting and relatively novel. Although uncommon, scenarios where pretrained models are accessible only through their predictions, without training data or model parameters, may arise in practice. The proposed features for estimating predictor utility are also interesting.

Critical weaknesses
  1. Limited methodological novelty. When the original training data and model parameters are inaccessible, the outputs of frozen predictors effectively serve as molecular features. AssayRouter is therefore essentially a feature selection method. While the engineered features are interesting, the meta-model itself is a standard histogram gradient boosting regressor.
  2. Marginal improvements and unclear computational benefits. The performance improvement over Support-CV is relatively small. The computational efficiency is achieved by amortizing substantial historical training costs, and it is unclear how significant this benefit is in a setting with very limited target data. Furthermore, the method uses frozen predictor outputs in a relatively simple way, without exhaustively exploring alternative approaches.
Questions for authors

How does AssayRouter compare with Tabular Foundation Models in predictive performance?

1 comment
AuthorsOpenReview ↗

Post-fit supervision, predictive gains and measured cost

We address your three concerns (the supervision principle, predictive value and cost) with new controlled comparisons. The core finding: the same features and HGB capacity, trained on post-fit utility labels instead of standalone source-quality labels, lower strict NLL from 2.2575 to 2.1102, and the method is faster than Support-CV on most workloads.

What we learn from completed assays

We select a frozen source for the improvement it can make to the local predictor. A source that exactly reproduces the local prediction can be accurate on its own and add zero in their convex combination. Completed assays let us measure this added value directly. On C, we fit the local-only predictor and a convex combination of the local predictor with one candidate. The candidate's utility is the first fit's loss minus the second fit's loss on the same held-out discovery molecules. AssayRouter-C subtracts the mean utility across candidates in each historical block, preserving their order (Eqs. 3–7).

On a new assay, we can already observe how a source behaves on the labeled routing molecules. Our 11 candidate features describe its alignment with the local model's out-of-fold residuals, its output distribution, disagreement among its frozen members and the available sample sizes. A shared HGB regressor learns how these measurements jointly predict post-fit utility. It ranks sources once per routing split. Separate labels fit the weights of the local predictor and the four selected sources.

Collaborative pQSAR already uses assay-model predictions as molecular descriptors (Martin and Zhu, 2021, Appendix A.2.1). Our source-quality control tests what the new supervision contributes. Its label measures how much a calibrated source improves on a constant predictor. We retain the 11 inputs, interface indicators, HGB capacity search, assay-grouped folds and final combiner. On 24 external targets, strict NLL is 2.2575 with source-quality supervision and 2.1102 with post-fit supervision (Table 1, Appendices A.1.2 and A.1.13). Shuffling candidate–utility labels within each historical block preserves the features and label distribution and raises strict NLL by 0.1352 [0.0852, 0.1915] (§4.1). Table 15 also evaluates greedy, query-local and full-bank alternatives.

Predictive performance and TabPFN

With the same four-source allowance and final fit, AssayRouter-C lowers strict NLL from 2.1511 to 2.1102 relative to Support-CV. It wins in 77.8% of target–interface comparisons. Mean absolute error falls from 26.24 to 23.39, a 10.9% reduction (Table 1). The strict NLL gains are 0.0736 and 0.0413 at 16 and 32 labels, respectively (Table 23).

We tested TabPFN-3.5 (package v9.2.0) on the same targets and supports. With the same 1,024 fingerprint coordinates and all support labels, TabPFN scores 2.2171 NLL and Ridge scores 2.3558. AssayRouter-C scores 2.1102 using four frozen source outputs and half the labels for its final fit. Its paired advantage over fingerprint TabPFN is 0.1068 [0.0206, 0.2037]. All methods use the paper's support-standardized Student-t loss for point predictions.

We also supplied TabPFN with frozen source outputs. Each fit uses the same cross-fitted local prediction column and the same half of the support labels.

Sources supplied to TabPFNStrict NLLPrediction-averaged NLL
All sources (5 in Biogen, 8 in ExpansionRx, 21 in TDC)2.27322.2343
Four selected by AssayRouter-C2.33122.2994
Four selected by Support-CV2.31112.2764

With TabPFN as the combiner, Support-CV's four sources yield lower strict NLL by 0.0202 [0.0051, 0.0377]. The historical labels measure the value of a source after the submitted Ridge/NNLS fit. That fitting procedure is where we observe the selection advantage.

Cost on a new assay

Across 432 target–interface–budget combinations, the median time to build the 16-constituent predictor and evaluate its confirmation molecules was 0.3085 s for AssayRouter-C, 0.6164 s for Support-CV and 0.5294 s for Support-greedy. Timing covers feature construction, scoring, selection and fitting on one CPU thread (three repetitions). Support-CV was faster on Biogen's five-source bank (0.2172 s versus 0.2573 s for AssayRouter-C).

Historical label construction and router training took 782.1 s. The mean saving of 0.3093 s against Support-CV implies a break-even of 2,529 predictor builds on this workload. For a single routing direction, the medians were 25.49 ms versus 43.50 ms. The per-direction saving of 12.27 ms gives a crossover of 63,762 directions. Models and representations were preloaded. Four families in Biogen and ExpansionRx execute fitted heads. The two neural families and all TDC interfaces read precomputed outputs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.