- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes an uncertainty-conditioned path-energy framework for mapping genomic sequence predictions and calibrated predictive uncertainty to distributions over phenotype hypotheses in biomedical knowledge graphs. The method assigns an interpretable energy to admissible gene–disease–phenotype paths, uses uncertainty-dependent temperature scaling to control distributional breadth, aggregates path probabilities into phenotype-level hypotheses, and implements a Hastings-corrected Metropolis–Hastings sampler whose stationary distribution can be checked against an exactly normalized target. The authors evaluate the framework on AlzKB and Hetionet. They report strong correlations between upstream uncertainty and phenotype-level entropy, verify that the sampler reproduces the exact target distribution, and evaluate masked disease–symptom edge recovery under gene-held-out settings. The proposed method achieves higher average recall@10 than DistMult across three graph masks and lower across-mask variability.
Critical strengths
The paper clearly separates the learned target distribution from the fidelity of the sampling operator. Exact enumeration allows the authors to verify the Hastings-corrected sampler directly, which is a useful methodological check.
The framework provides an interpretable mechanism for propagating upstream uncertainty into the breadth of graph-supported hypotheses. The use of path-level features and phenotype-level aggregation makes the output more inspectable than a black-box ranking score.
Critical weaknesses
The central uncertainty result is largely imposed by the model parameterization rather than demonstrated as calibrated phenotype uncertainty. The temperature is explicitly defined as an increasing function of the supplied uncertainty, and the paper proves that increasing temperature increases entropy for fixed utilities. The high uncertainty–entropy correlations therefore mainly confirm the designed coupling; they do not establish that the resulting phenotype distributions are calibrated against phenotype-level uncertainty or clinical outcomes.
The biological evaluation is based on a small and highly constrained setup. The AlzKB experiments use 982 variants from only 15 genes, with APP, PSEN1, and PSEN2 accounting for 94% of queries, and the energy is learned from only 121 definitive pathogenic/benign variants. The candidate supports are constructed from the same knowledge graphs used for inference, so the results do not establish generalization to unseen genes, diseases, phenotypes, or external patient-level data.
The evidence for improved recovery over DistMult is not robust across the reported splits. Under gene-held-out evaluation, the pooled recall@10 improvement is positive, but the improvement for mask seed 42 has a confidence interval containing zero. On the primary split, the full model and DistMult differ by only 0.013 recall@10 with an interval that also contains zero. Thus, the headline aggregate improvement is sensitive to how the graph is masked and aggregated.
The method’s contribution is difficult to isolate from the learned energy and candidate-support design. The full model includes route features, endpoint features, confidence interactions, and uncertainty-dependent temperature. Several fixed-temperature and component ablations achieve similar recovery, and the paper does not show that uncertainty conditioning itself improves held-out biological association recovery. The sampler, meanwhile, is not a computational improvement over direct categorical sampling because the candidate supports are enumerable.
Questions for authors
Can the authors evaluate phenotype-level probability calibration using independent phenotype or patient-level outcomes, rather than only entropy correlations and masked graph-edge recovery? Without such an evaluation, what is the intended interpretation of the output probabilities?
Can the authors report performance on gene-held-out, disease-held-out, and phenotype-held-out splits that prevent the model from relying on graph neighborhoods closely related to the training queries? The current evaluation leaves held-out genes in the graph and uses a small, highly concentrated set of genes.
How does the method perform when the upstream sequence uncertainty is deliberately miscalibrated or perturbed? Since the uncertainty-to-temperature schedule is hand-specified, a robustness analysis would clarify whether the framework responds to uncertainty quality or merely to the numerical value of H.
Can the authors provide per-mask and per-gene statistical results, rather than relying primarily on pooled query-level averages? In particular, the non-significant improvement on one of the three masks and the wide variation in the number of reachable paths should be discussed more prominently.
Can the authors isolate the contributions of uncertainty conditioning, route features, and the learned energy through matched ablations evaluated on the same held-out recovery task? The current results show that the full framework works better in aggregate, but do not establish which component is responsible for the improvement.
Since the candidate supports are enumerable, please clarify the practical motivation for the Metropolis–Hastings sampler. What computational regime or graph size would make sampling preferable to exact normalization, and how does the method scale when enumeration is no longer feasible?
3 comments
W1 / Q1. Calibration. Agreed, and this is not claimed. Proposition 1 covers path entropy at fixed utilities; that phenotype entropy still tracks H after aggregation is empirical and holds within every gene (median within-gene ρ 0.96 / 0.90) and without the three dominant genes (0.950 / 0.769; R1). It is not calibration: under a log score on held-out positives the conditioned distribution is not better than fixed temperature (−10.99 vs −8.32 bits). No patient-level data exist for these variants. The outputs are relative support among graph-supported hypotheses, as §3.3 states.
W2 / Q2. Hold-out structure. Held-out genes must stay as nodes or no path exists; all their labels are withheld from calibration and energy fitting. New disease hold-out (all symptom edges of 20% of diseases removed, five folds, combined with gene hold-out; R6): recall@10 0.108 vs chance 0.056, +0.052 [+0.025, +0.088]; not different from DistMult (+0.013 [−0.127, +0.032]). A phenotype with no remaining edges is unreachable for any graph-supported method, so phenotype hold-out is outside the method's scope; we say so.
Q3. Miscalibrated H (Table R5). Breadth follows the value supplied: ρ with the true H is 0.74 with the true signal, 0.68 with added noise (σ = 0.2), 0.17 when H is permuted across variants, −0.58 when inverted. Recall@10 is unchanged (0.166–0.182). The framework transmits H faithfully; it cannot repair a poor H, and on locked likely-label variants our upstream H flags sequence-call errors only weakly (AUROC 0.56). We add this as a limitation.
Q6. Why MH. As §3.4 states, there is no computational benefit on enumerable supports; the sampler is included so it can be verified against an exact target. Uncapped supports grow about 100× per gene hop (median 85, 4,980, 6.0×10⁵, 7.3×10⁷ paths for 0–3 hops; R7), so enumeration fails beyond one hop. Our current proposal is a global independence proposal and would itself need replacing by a local or amortized one in that regime; the Hastings correction and the exact-reference test carry over. We make no scalability claim.
Dear Reviewers,
Thank you again for your thoughtful comments. Please let us know whether our responses have addressed your concerns or if any further clarification is needed.
Reply to Reviewer 6kjL
Thank you for the detailed review. We ran every analysis you asked for that our data allow (Tables R1–R8 in our general comment; gene-held-out, gene-cluster 95% bootstrap intervals).
W3 / Q4. Robustness of the DistMult comparison; per-mask and per-gene statistics. Per-mask and per-gene results for ten masks are in Table R2a–c (pooled difference +0.053 [−0.038, +0.086] query-weighted, +0.016 [−0.049, +0.091] gene-balanced). To separate mask effects from DistMult's training noise we then ran a confirmation test with the protocol fixed in advance: 30 fresh masks, DistMult averaged over three training seeds (Table R8 in the general comment):
Narrower claim. Query-weighted recall@10 is higher than DistMult's on most masks (24 of 30), an advantage concentrated in APP, PSEN1 and PSEN2 (0.191 / 0.197 / 0.110 vs 0.132 / 0.094 / 0.097); the gene-cluster interval includes zero and the other 12 genes split 6–6, so we do not claim an advantage across genes. Across-mask CV is lower for our method query-weighted (0.30 vs 0.39) and higher gene-balanced (0.16 vs 0.12), so we no longer claim lower variability. Both statements will replace the current sentence in the abstract and Section 5.7, with gene-balanced and per-gene results reported next to query-weighted ones.
Why the paper's conclusions do not rest on exceeding DistMult. DistMult is a reference for task difficulty: a trained triple scorer that receives neither the sequence prediction nor its uncertainty and returns a score, not a distribution over traceable paths. The contribution is the uncertainty-conditioned interface, and each part of it is supported independently of that comparison: (i) hypothesis breadth tracks upstream uncertainty within every gene with ≥5 variants and without the three large genes, on both graphs (Table R1); (ii) the target is exact and the sampler is verified against it; (iii) training under θ(H) improves held-out recovery over the same energy retrained at constant temperatures from 0.20 to 0.80, by 0.043–0.061 with intervals excluding zero (Table R3); (iv) recovery is above chance under both weightings on all 30 fresh masks (+0.118 [+0.060, +0.141]; +0.095 [+0.037, +0.167]) and under disease hold-out (Table R6). The method does this while remaining at least on par with the trained embedding.
W4 / Q5. What the contribution is, and which component is responsible for what. The novelty is not a new energy family or a new sampler; both are standard, as Sections 2 and 3.4 say. It is the interface: an upstream model's calibrated uncertainty about a query sets the concentration of an exactly normalized distribution over graph-supported phenotype hypotheses, and that distribution, not a ranking score, is the evaluated output. The matched ablations you asked for, on the same held-out recovery task (ten masks, identical queries; Tables R3 and R4), assign a role to each component:
So uncertainty conditioning is responsible for the breadth behaviour and, when the energy is trained under it, for a recovery gain over matched constant temperatures; the submitted "fixed θ" ablation changed the temperature only at inference, which is why it showed no difference. Hotter constants (1.00, 1.25) come close (+0.022 [−0.003, +0.040]; +0.015 [−0.008, +0.033]), and we will state that. The sampler is not claimed as a contribution: it is the component that would be needed beyond enumerable supports, included here because it can be verified against the exact target.