- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
P2R-LLM studies prediction of GDSC2 drug responses for held-out drugs or cell lines, with either zero or five measured responses from the unseen entity. It combines frozen molecular and expression features, textual biological facts, and optional support measurements in a Qwen3-14B language model. Training proceeds through response-free biological alignment, teacher-guided reasoning warmup, and alternating numerical prediction and REINFORCE updates. A 100-token response interface converts the reasoning-conditioned hidden state into a continuous LN IC50 estimate. The method achieves the lowest entity-macro RMSE among the reported baselines in all four evaluation settings, with further cell-blind component ablations.
Critical strengths
- A carefully specified entity-held-out evaluation. Drug-blind and cell-blind protocols are reported separately, with zero- and five-measurement settings, fixed support panels, explicit query exclusions, and train-fitted response bins (Sections 3.1, 3.3, and 4; Appendix A). These details make the claimed operating regime much easier to assess than a generic random drug–cell pair split.
- Competitive prediction results with relevant controls. Tables 1–2 report the best RMSE in all four comparisons, including 1.6109 versus 1.7116 in drug-blind zero-shot prediction. The model also improves from 1.6109 to 1.2632 when five same-drug responses are supplied. Table 3 retains a direct no-CoT predictor with the same biological and support inputs, which is particularly useful for judging what the trained reasoning pathway adds.
Critical weaknesses
- The evidence does not yet isolate the value of the reasoning content. In Table 3, the no-CoT model already reaches 1.2461 cell-blind RMSE at k = 0, versus 1.3261 for MLP; generated reasoning lowers this further to 1.2337. The authors correctly note that this ablation changes intermediate autoregressive computation as well as text content. Likewise, the w/o RL control removes prediction-error feedback together with format and length rewards, so it cannot isolate the contribution of outcome-based reasoning optimization. The experiments support the complete training pathway, but leave unclear whether the biological comparisons expressed in the trace are needed for the prediction improvements.
- The margins and comparison design leave the headline generalization advantage uncertain. In drug-blind five-shot prediction, the improvement over the strongest baseline is only 1.2753 to 1.2632 (0.95%; Table 2). Appendix B reports one training seed per protocol and a fixed support panel, without uncertainty across seeds, support draws, or held-out entities. The baselines legitimately use the same support measurements, but typically adapt via residual KRR and do not share the pretrained language backbone or textual facts (Section 4; Appendix C). These comparisons establish a promising full-system result, not a controlled advantage of in-context reasoning over simpler support-aware predictors. More evidence is needed before treating the small reported gaps as robust.
Questions for authors
- Are there existing evaluations in which generated traces are replaced, shuffled, or otherwise perturbed while the same model, biological inputs, and approximate inference computation are retained? If not, which result most directly supports the need for semantically meaningful intermediate text?
- Does the reported w/o RL result include removal of both the response-error term and the format/length terms in Eqs. (7)–(8)? Are any existing separate-component results available?
- For the drug-blind k = 5 comparison, can the authors report paired per-drug RMSE differences, or any existing uncertainty estimates across held-out drugs or support-panel selections? How many of the 24 test drugs improve over the strongest baseline?
- Is a same-input direct predictor with a simple support-based calibration available beyond the no-CoT control, and are there any drug-blind component ablations? These would help distinguish in-context adaptation from the effect of the pretrained representations and the broader training recipe.
6 comments
Thank you for the additional analyses. The trace-replacement experiment, drug-blind ablations, and calibrated direct-prediction comparison provide useful additional evidence. However, the trace experiment does not establish the biological validity of the generated reasoning, and the small RMSE advantage over the calibrated direct predictor remains uncertain without repeated seeds or support-panel resampling. The description of the No-RL ablation and RL update frequency also needs clarification. I therefore maintain my original assessment.
Thank you for the follow-up. We address the evidence for the content of the generated reasoning, the stability of the comparison with the calibrated direct predictor, and the distinction between the No-RL controls and the full training schedule.
1. What the reasoning contributes
We understand the concern about whether the generated reasoning contains biologically meaningful inferences. We found two zero-shot examples involving drugs held out of response training in which the model inferred downstream effects consistent with published experiments in the same cell lines. For BMS-345541 in SK-MEL-5, the trace linked IKK inhibition to reduced NF-κB signaling, consistent with the decrease in NF-κB activity measured by Yang et al. [1]. For UMI-77 in BxPC-3, the trace linked MCL1 inhibition to the release of pro-apoptotic proteins, consistent with the disruption of MCL1 binding to BAX and BAK reported by Abulwerdi et al. [2]. In both cases, the model used the supplied target annotation to generate a testable hypothesis about a downstream effect.
To test whether predictions respond to the direction of that biological relationship, we selected several zero-shot drug-blind queries () and made two edits to each generated trace. One rewrites the mechanism in different words while preserving its meaning. The other uses the same rewritten sentence structure and reverses the direction of the effect. We held the checkpoint, query, biological inputs, and surrounding reasoning fixed. The two edits preserve the same factual observations and have equal token counts, so their comparison tests sensitivity to the direction of the stated mechanism.
BMS-345541 in SK-MEL-5
Original trace
BMS-345541 is an IKK inhibitor, and CHUK expression is 4.1 log2(TPM+1) in this melanoma. No measured supports are available; expression alone does not establish sensitivity. IKK inhibition could reduce NF-κB signaling, but the conditional expectation remains uncertain.
Meaning-preserving rewrite
BMS-345541 is an IKK inhibitor, and CHUK expression is 4.1 log2(TPM+1) in this melanoma. No measured supports are available; expression alone does not establish sensitivity. When IKK is blocked, the NF-κB pathway could become less active. The resulting sensitivity in this cell line remains uncertain.
Reversed mechanism
BMS-345541 is an IKK inhibitor, and CHUK expression is 4.1 log2(TPM+1) in this melanoma. No measured supports are available; expression alone does not establish sensitivity. When IKK is blocked, the NF-κB pathway could become more active. The resulting sensitivity in this cell line remains uncertain.
| Reasoning text | RMSE on selected queries (log IC50) ↓ |
|---|---|
| Original trace | 1.4607 |
| Meaning-preserving rewrite | 1.4671 |
| Reversed mechanism | 1.4812 |
We further extended this edit-based analysis to a total of eight samples: the two examples with explicit evidence support described above, plus six samples randomly selected from cases whose reasoning sentences contained clear mechanistic statements. For the meaning-preserving rewrites, RMSE decreased on 4 samples and increased on 4 samples, with the overall RMSE remaining essentially unchanged (only 0.0064 higher on average). In contrast, for the mechanism-reversed edits, RMSE increased on 6 samples and decreased on 2 samples, with an average increase of 0.0205. These results indicate that our reasoning is direction-sensitive and carries genuine inferential meaning, rather than merely superficial paraphrasing, and thus has reference value for real biological research.
References
[1] Yang J, Amiri KI, Burke JR, Schmid JA, Richmond A. BMS-345541 targets inhibitor of κB kinase and induces apoptosis in melanoma: involvement of nuclear factor κB and mitochondria pathways. Clin Cancer Res. 2006;12(3 Pt 1):950–960. DOI
[2] Abulwerdi F, Liao C, Liu M, et al. A novel small-molecule inhibitor of Mcl-1 blocks pancreatic cancer growth in vitro and in vivo. Mol Cancer Ther. 2014;13(3):565–575. DOI
Thank you for asking us to clarify the role of the reasoning text and the reward. We report the trace substitutions, define the RL ablation, and give the paired drug results and the comparison with a direct predictor.
1. Trace replacement at a fixed checkpoint
We replaced the generated trace with a similarly long trace from a different drug on the same drug-blind development queries, keeping the model, biological inputs, and supports fixed.
At k=5, shuffling raises Micro RMSE by 0.1133 (95% paired interval [0.0197, 0.2330]). The original trace has lower error on 15 of 24 drugs, showing that the predictor responds to whether the trace fits the query.
In the separate drug-blind training ablation, No-CoT scores 1.8360 / 1.4839 Macro RMSE at k=0/5, versus 1.6908 / 1.4315 for No-RL. The ablation changes the training pathway, while shuffling changes the text at fixed weights.
2. What the w/o RL ablation removes
Yes. The Table 3 w/o RL ablation removes the entire REINFORCE term, including prediction-error feedback, the completion bonus, and the length penalty. It retains teacher warmup, trace sampling, and numerical supervision on sampled traces.
RL accounts for roughly one tenth of the response-training updates up to the evaluated drug-blind checkpoint. Isolating the effects of individual reward terms at this share would require repeated runs. Time and compute precluded those runs during rebuttal. We will add a detailed sensitivity analysis of the loss weights and reward terms in the revised manuscript.
3. Per-drug differences and paired uncertainty
In the drug-blind table at k=5, our Macro RMSE is 1.2632 versus 1.2753 for DeepCDR. Our model has lower error on 15 of 24 drugs than DeepCDR. The table gives the paired differences for all 24 drugs against DeepCDRat k=5. A positive difference means our model has lower RMSE.
Similarly, compared with TopDr, our model shows an advantage on 16 out of 24 drugs, and compared with PASO, it shows an advantage on 18 out of 24 drugs. This indicates that although the numerical RMSE advantage of our method in the drug-blind setting is not substantial, our method maintains a consistent advantage across different drugs.
4. Direct prediction and drug-blind ablations
The full model has the lowest Macro RMSE at both support levels in the drug-blind ablations.
Compared with No-RL, CoT and biological alignment each lower error. CoT links evidence to the prediction, biological alignment helps the model read molecular and expression features, and RL guides trace generation using prediction error.
We used the baselines' residual kernel ridge regression to calibrate the direct predictor using five support measurements per drug.Simple calibration also substantially reduces Macro RMSE (from 1.6109 to 1.2707), but it remains inferior to our method (1.2632). Moreover, simple calibration yields only limited PCC improvement (0.5779), consistent with the baseline results in the main table, whereas our ICL-based approach improves PCC to 0.6009.