- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies a new method for adaptation of pretrained ligand-target affinity prediction on larger macro-molecules. The adaptation assumes that no direct accesses to the original source data used for pretraining but a new source data is available but practically different from the original source and another target macro-molecule data is available.
The main idea behind the adaptation is to rerank the contribution of the families in the new source data to penalise family with highest degradation loss while ignoring the families that the degradation do not happen. The idea is similar GRPO but applied in the context of the domain adaptation problem.
Critical strengths
The proposed idea is interesting and may work in the context of continue learning. The comparison study is strong with strong baselines and relevant experiments, I have a concern on this perspective please see in the weakness points. The paper was well written and easy to follow.
Critical weaknesses
My main concern is that the proposed approach trade old model retention for adaptation to new macro-molecule so the hyperparameters used to balance the losses have an important impact. In the experimental settings the proposed approach used tuned hyper parameters but it is unclear whether all the baseline approaches which have hyperparams were tuned carefully. I therefore recommend a weak reject but if the authors provide additional experiments as I request below and the results are still significant I will reconsider my recommendation:
-
Please provide the results of the baseline including Uniform Replay, GRPO when their hyper params are finetuned carefully with the same setting as the proposed method.
-
Please provide the results of finetuned method not only with macro molecules but also finetuned on the new source data because the given method is not finetuned on larger data as the other baseline.
-
Please varied the hyperparams of all methods and plot the pareto frontiers according to two metrics: one on the target molecules and one on the new source molecules, report the Hyper Volume of each method, if the proposed method can find better HV solution that would be a definite strong results.
Questions for authors
Please look at the questions I provided in the weaknesses points.
8 comments
Responses to Reviewer 6214 (2/4)
Q1: “Please provide the results of the baseline including Uniform Replay, GRPO when their hyper params are finetuned carefully with the same setting as the proposed method.”
A2: Thank you for asking for this. We swept the two methods the question names, uniform replay and group DRO with the absolute signal, and extended the same sweep to every other baseline of Table 3 that has a tunable strength, so the comparison is matched across the whole table rather than for two methods alone. Every baseline with a tunable strength is now swept over three random seeds under the identical protocol, source stream, 1:1 draw, target exposure and step budget as RPR, with the same protocol IV checkpoint rule. Methods appear in the order of Table 3, and the third column marks the configuration that appears in the submitted Table 3. Group DRO with the absolute q signal is the ablation of Table 4 rather than a Table 3 baseline; we sweep it alongside the others because it shares our grouping, stream and machinery and differs only in using an absolute rather than a reference-relative weight signal.
| Method | Configuration | In Table 3 | Source Fam. ρ ↑ | Target Fam. ρ ↑ |
|---|---|---|---|---|
| Pretrained Boltz-2 (no FT) | n/a | ✓ | 0.356 | 0.050 |
| FT | no tunable strength | ✓ | 0.276 ± 0.043 | 0.177 ± 0.034 |
| L²-SP | λ = 0.1 | 0.276 ± 0.068 | 0.201 ± 0.027 | |
| L²-SP | λ = 1.0 | ✓ | 0.279 ± 0.042 | 0.186 ± 0.030 |
| L²-SP | λ = 10 | 0.340 ± 0.005 | 0.146 ± 0.038 | |
| EWC | λ = 0.1 | 0.272 ± 0.106 | 0.228 ± 0.108 | |
| EWC | λ = 1.0 | 0.291 ± 0.101 | 0.206 ± 0.017 | |
| EWC | λ = 2.0 | ✓ | 0.351 ± 0.028 | 0.182 ± 0.033 |
| OD | λ = 0.3 | 0.409 ± 0.078 | 0.188 ± 0.032 | |
| OD | λ = 1.0 | ✓ | 0.404 ± 0.049 | 0.184 ± 0.043 |
| OD | λ = 3.0 | 0.355 ± 0.032 | 0.199 ± 0.028 | |
| UR | source:target 1:3 | 0.434 ± 0.014 | 0.213 ± 0.089 | |
| UR | source:target 3:5 | 0.394 ± 0.060 | 0.147 ± 0.013 | |
| UR | source:target 1:1 | ✓ | 0.384 ± 0.068 | 0.171 ± 0.050 |
| AGEM | mem = 8 | 0.338 ± 0.058 | 0.208 ± 0.012 | |
| AGEM | mem = 16 | 0.386 ± 0.028 | 0.133 ± 0.052 | |
| AGEM | mem = 32 | ✓ | 0.429 ± 0.138 | 0.168 ± 0.063 |
| Group DRO (absolute q signal) | η = 0.1 | 0.417 ± 0.069 | 0.213 ± 0.110 | |
| Group DRO (absolute q signal) | η = 0.4 | 0.325 ± 0.031 | 0.123 ± 0.047 | |
| Group DRO (absolute q signal) | η = 1.0 | 0.405 ± 0.042 | 0.188 ± 0.092 | |
| RPR (ours) | η = 0.4, source:target 1:1 | ✓ | 0.455 ± 0.042 | 0.261 ± 0.013 |
No configuration reproduces the trade-off. On the target axis nothing reaches RPR's 0.261; the closest is EWC at λ = 0.1 with 0.228 ± 0.108, and RPR's worst of three seeds, 0.253, is still above the mean of every other configuration in the table. No configuration improves on RPR on the two axes together, and RPR attains its best value on both axes at one and the same setting rather than needing a different one per metric.
Matched tuning also answers how much of the spread is tuning and how much is noise. The seed-to-seed variation of the baselines is an order of magnitude larger than ours: on the target axis EWC at λ = 0.1 ranges from 0.104 to 0.298 over the three seeds and absolute-signal Group DRO at η = 0.1 from 0.114 to 0.331, against 0.253 to 0.276 for RPR. A single run of a baseline is therefore close to unreadable, which is why the default settings quoted in the submission looked weaker than they should, and it is the reason we report every cell as a three-seed mean here. Two of the axes are not even monotone in their own hyperparameter: AGEM gives 0.208, 0.133 and 0.168 on the target axis at mem = 8, 16 and 32, and UR gives 0.213, 0.147 and 0.171 at 1:3, 3:5 and 1:1, so neither offers a direction in which a better setting could be extrapolated.
Retention measured where it matters keeps the same ordering. Aggregate retention can rise while individual families deteriorate, so we also compare the upper tail of the per-family degradation on 1,909 held-out source records and 36 families. RPR's worst family degrades by 0.226 ± 0.043 and its second-worst by 0.174, against 0.442 ± 0.132 and 0.328 for uniform replay at the submitted fraction, which sees the identical source stream; RPR is lower in all three seeds.
Responses to Reviewer 6214 (3/4)
Q2: “Please provide the results of finetuned method not only with macro molecules but also finetuned on the new source data because the given method is not finetuned on larger data as the other baseline.”
A3: Thank you for catching this confound. The comparison is already available in the paper, because uniform replay is fine-tuning on the union of the two streams. UR applies the plain Boltz-2 affinity loss to target and source records with no retention term, no reweighting and no reference model, differing from FT in exactly one respect, that it also draws source batches. Reading the three rows in that light gives the decomposition the reviewer asks for:
| Method | Data seen | Source Fam. ρ ↑ | Source ρ ↑ | Target Fam. ρ ↑ | Target ρ ↑ |
|---|---|---|---|---|---|
| Pretrained Boltz-2 (no FT) | n/a | 0.356 | 0.455 | 0.050 | 0.283 |
| FT (target only) | 15,516 | 0.276 ± 0.043 | 0.462 ± 0.016 | 0.177 ± 0.034 | 0.262 ± 0.064 |
| UR = FT on target ∪ source | 35,509 | 0.384 ± 0.068 | 0.567 ± 0.053 | 0.171 ± 0.050 | 0.265 ± 0.044 |
| RPR (ours) | 35,509 | 0.455 ± 0.042 | 0.603 ± 0.012 | 0.261 ± 0.013 | 0.310 ± 0.029 |
Adding the source data recovers retention; reweighting it is what improves both sides. Adding the source data (FT → UR) recovers small-molecule retention, from 0.276 to 0.384 on Source Fam. ρ and from 0.462 to 0.567 on Source ρ, while the macrocycle side does not improve at all and in fact moves from 0.177 to 0.171. More data by itself buys retention and nothing else. Reference-relative reweighting (UR → RPR) then improves both sides at constant data volume: Source Fam. ρ 0.384 → 0.455 and Target Fam. ρ 0.171 → 0.261, with the identical stream, the identical target exposure and the identical step count. The second comparison is the paper's claim, and it is measured at equal data, which is the control the question asks for.
On the mixing proportion, UR draws the two streams at source:target 1:1, whereas their natural proportion would put slightly more weight on the source; the sweep in A2 varies that ratio and no setting of it gives UR a simultaneous advantage on the two axes.
Responses to Reviewer 6214 (1/4)
We sincerely thank the reviewer for stating so clearly what additional evidence would be useful. Your points on tuning the baselines under the same setting, on fine-tuning with the union of the two data streams, and on presenting the trade-off as a Pareto frontier with hypervolume have been extremely helpful, and we have run all three. We summarize the issues raised and address them one by one below. Numbers are mean ± sd over three random seeds; bold marks the best method in each column.
W1: “My main concern is that the proposed approach trade old model retention for adaptation to new macro-molecule so the hyperparameters used to balance the losses have an important impact. In the experimental settings the proposed approach used tuned hyper parameters but it is unclear whether all the baseline approaches which have hyperparams were tuned carefully.”
A1: Thank you for this concern. Two things answer it: the baselines are now tuned under our own protocol, and at our operating point the trade-off does not have to be paid.
On the tuning. Every baseline with a tunable hyperparameter is now swept under the identical setting as RPR and over three random seeds (A2). On our own side two settings were tuned, the step size η and the source:target draw, both chosen on the target validation set by the same median within-family Spearman statistic that selects checkpoints, from the sweeps the paper reports in Table 6 and Table 5; every other setting was fixed in advance.
The gain on macrocycles is not paid for on the small-molecule side. RPR improves on the released model on both axes by much the largest margin: Source Fam. ρ 0.455 ± 0.042 against 0.356 for pretrained Boltz-2, and Target Fam. ρ 0.261 ± 0.013 against 0.050, which is +0.099 and +0.211 where the next best margins are +0.073 and +0.134. The quantity the trade-off is supposed to protect is protected as well: on 1,909 held-out source records and 36 families, the largest per-family degradation relative to the pretrained model is 0.226 ± 0.043 for RPR against 0.442 ± 0.132 for UR, lower in all three seeds.
The balance is not fragile in η. At η = 0.1 / 0.4 / 1.0 the three-seed means are Source Fam. ρ 0.394 / 0.455 / 0.392 and Target Fam. ρ 0.218 / 0.261 / 0.197, every one of them above UR's 0.384 and 0.171, while collapsing the family weights to uniform (η = 0) gives 0.419 / 0.138 and loses the target-side gain. That isolates reference-relative reweighting, rather than a particular loss balance, as the component doing the work.