- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces HRD-BioAgent, a framework designed to predict homologous recombination deficiency (HRD) status—defined by genomic scar scores—by integrating data from whole-slide pathology images, transcriptomics, mutations, and methylation profiles. The framework employs an "HRD Evidence Map" to track the provenance, target population, and type of evidence; utilizes conditional tool invocation to incorporate supplementary information, such as SBS3 mutational signatures; and generates probability estimates via a calibrated statistical meta-model trained using an in-fold approach. The large language model (LLM) serves to orchestrate the workflow and synthesize textual output, rather than directly generating numerical predictions.
Critical strengths
1.The paper explicitly distinguishes between empirical data, model predictions, and secondary inferences; it delegates the generation of numerical probabilities to calibrated statistical models while maintaining a record of evidence sources and missing data. This design facilitates auditing and prevents the misinterpretation of unmeasured modalities as negative results. Deterministic checkers provide concrete, testable safeguards against process errors.
2.Tables 2 and 3 compare "stacking," static SBS3 enhancement, and the full system using the same set of patients, explicitly acknowledging that the final incremental step did not reach statistical significance. Table 7 also reports competitive results using L1 feature selection, without interpreting these as an advantage in agent reasoning. These analyses provide a useful basis for evaluating the sources of performance.
Critical weaknesses
-
The most distinct performance gain in Table 3 arises from the static inclusion of SBS3—a mutational signature with well-established utility in predicting homologous recombination deficiency (HRD), as seen in tools like HRDetect. Since the evidence features utilized by the full system differ from those used in the static SBS3-only baseline, the observed improvement cannot be attributed solely to the reasoning process or the orchestration logic. Furthermore, the L1 baseline in Table 7 achieves performance comparable to that of the human-curated evidence set. The adaptive strategy described in Appendix J—which determines whether to escalate an analysis based on fixed probability thresholds—has not been benchmarked against a standard deterministic cascade (i.e., one employing the same inputs, tools, calibration methods, and thresholds). Moreover, the paper lacks sufficient documentation regarding the specific LLMs employed, the prompts used, the selection of tools, and the operational stability of the system. Consequently, it remains unclear whether the language model actively influences decision-making or if the same outcomes could be achieved through standard stacking combined with rule-based routing. The "85/85" error detection metric primarily validates the execution of predefined checks, rather than the biological accuracy of the synthesized textual evidence. For a paper that positions itself primarily around the concept of "agents," this represents a significant gap in the clear attribution of its core contributions.
-
Sections 3.2–3.3 require that evidence estimation and feature selection be restricted to the training folds; however, Section 5 explicitly acknowledges that the priors for BRCA-related evidence were not entirely determined within the outer training folds. It must be clarified which features or thresholds utilized labels from the full cohort, and whether the primary findings remain robust under strict data isolation. Furthermore, a mere statement that "out-of-fold" (OOF) predictions were used is insufficient to verify the complete nested procedure involving base models, meta-models, and isotonic calibration. There is also a direct contradiction in the rules for "concordance" analysis: Section 4.5 of the main text requires at least two out of the four pieces of evidence to be positive, whereas Appendix A modifies this rule to require at least one positive piece of evidence excluding TP53—while simultaneously noting that only 25 of the original 105 "Concordant+" cases remain classified as such. Under the aforementioned definition, any patient who met the "two positive pieces of evidence" criterion must have had at least one piece of evidence other than TP53; therefore, assuming the predictions remained unchanged, such cases should not have lost their "concordant" status. Additionally, Figure 3 labels a case as "Concordant+" even though it lacks mutations in TP53, BRCA1, or BRCA2, has no methylation data, and only meets the E2F criterion—a classification that conflicts with the rules stated in the main text. Given that "auditable concordance assessments" constitute the core output of this paper, these discrepancies must be resolved through the establishment of consistent rules, the presentation of patient-level results, and the provision of an executable implementation.
Questions for authors
1.Please report the LLM used, its version, the prompts employed, and the tool-calling mechanism; furthermore, specify which actions are determined by the LLM and which are governed by fixed rules. Given identical evidence (across 11 items) and training protocols, how do static meta-models and deterministic cascades compare to a full agent in terms of predictions, invocation costs, and reporting accuracy?
2.Which BRCA evidence priors or thresholds utilize labels from patients in the hold-out test set? Please provide the complete nested pseudocode—covering the base model, meta-model, calibrator, and the selection of routing thresholds—and report the primary results obtained under strict isolation.
3.Please explain how the reduction from 105 cases to 25 cases in Appendix A is derived from the stated rules, and clarify why Figure 3 is labeled "Concordant+." Were different rules applied to Figure 3, Table 5, and Appendix A? How would the group sizes and conclusions change if a consistent set of rules were applied throughout?
4.In Table 13, when a specific modality is removed, are all evidence features derived from that modality also removed—rather than just the predictions from the corresponding base models? How do the multi-stage models described in Appendix J handle evidence that has not yet been acquired, and are the routing thresholds determined solely based on the training data? Please also provide information regarding the uncertainty stemming from differences in strategies, as well as the actual invocation rate of SBS3.
2 comments
Thanks for your rebuttal. However, I believe this article has room for further revision, and you failed to write the AI usage instructions as required by ICLR, which is a major oversight. I am maintaining my original score.
Response to Reviewer F8d7
Thank you. We accept your points. G1 to G4 refer to the general response; numbers refer to the revised PDF.
W1.
W2. Full-cohort information; nested procedure. In the submitted version, pathway selection and expression normalisation used the full cohort, and routing thresholds were fixed constants. The revised procedure is written out in Appendix A.4: (1) outer split; (2) unimodal learners fitted on the outer training partition, with out-of-fold scores for training rows; (3) knowledge features mapped without labels, direction check and z-scoring on training rows; (4) configuration chosen by inner 3-fold cross-validation; (5) penalty, calibration (isotonic if N ≥ 300, otherwise Platt) and sensitivity-90 threshold from inner out-of-fold predictions; (6) one refit, and the test fold scored once. A label guard recorded 0 violations. The residual dependence and its nested bound are in G2.
W3. Concordance contradiction; Figure 3. See G3. Patient-level results are in Table 21 and release states across cohorts in Table 23. The same code computes the release state for both tables and the case figure.
Q1. LLM details: W1(v). Predictions: W1(iv). Invocation cost: 905 spectrum calls