- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies the retrieval of intermediate mechanistic evidence in biomedical knowledge graphs when source and target are known. The paper proposed 3 stages: select candidate bridges from different biological views, estimating the utility probability of the bridges and select a subset, then control the bridge influence through a query specific gate. Lightweight supervised models are trained to estiamte the bridge utility probability and query-level reliability. The searching scenario is a beam search combining graph and bridge scores under a fixed a search budget.
On DrugMechDB, Table 1 reports Path Success increasing from 27.02% to 39.52% and Edge Recall from 33.97% to 59.46% over Graph Only. Further experiments cover candidate coverage, usefulness scoring, single-Bridge selection, and constant or shuffled gate controls.
Critical strengths
S1. The Stage 2 experiment provides useful evidence about candidate utility. In Table 2, the retrieval-aware scorer distinguishes useful Bridges within the clean subset, with AUROC/AUPRC of 0.815/0.655 compared with 0.515/0.257 for lexical similarity. The separate single-Bridge intervention also improves Edge Recall from 38.19% to 39.92% under matched retrieval conditions. These results support the value of learning downstream usefulness within the controlled setting.
S2. The main retrieval computation can be checked using the released artifacts. New graph searches reproduced the main DrugMechDB Clean results. Refitting the confidence and gate estimators from the released training tables also recovered the main retrieval outcomes.
Critical weaknesses
W1. The main experiment does not establish the complete candidate-construction and selection procedure.
W1.1 — The main results start from reference-aligned candidates, leaving the actual bridge selection process insufficiently validated. In the released clean test inputs, all 692 candidate bridges are nodes of their corresponding reference mechanisms, and every one of the 248 queries already has its intermediate answer entity among the candidates. These inputs allow a controlled study of how candidates guide search. However, without the reference-aligned clean candidates, whether the stage 1 can automatically select effective candidate is questionable.
W1.2 — The main comparison does not test the proposed stage 2 Bridge-selection rule. Section 4.3 describes greedy selection using c_h − μ Red(h,H) + β TypeBonus(h,H). However, every released validation/test candidate pool contains at most 3 distinct nodes, matching the Bridge budget of 3. The released main retrieval code keeps all of them, so Stage 2 does not filter the saved candidate sets.
The package's audit/DRUGMECHDB_SELECTOR_PROVENANCE.md explicitly states that the original DrugMechDB redundancy function, coefficient, and complete selector were not recovered.
Semantic+Fixed assigns each candidate confidence 1, while Utility+Fixed uses learned confidence. both use gate 0.3. This comparison tests confidence weighting, but the selected sets do not change.
W2. The experimental evidence does not sufficiently establish the contribution of the learned components in the main setting.
W2.1 — The Full results in Tables 1 and 4 are not sufficiently reconciled. Table 1 reports Full Edge Recall of 59.46%, whereas Table 4 reports 38.05% on the same held-out split. The released Clean Full row in final_continuous_by_condition.csv has mean gate 0.36035, the Full row in stage3_controls_summary.csv has mean gate 0.10917. Section 5.5 states that it fixes the selected Bridges to isolate semantic influence, but this does not explain the change in Full.
W2.2 — Most of the headline improvement is already obtained with fixed guidance from the supplied candidates. In Table 1, Semantic+Fixed reaches 39.11% Path Success and 58.56% Edge Recall, compared with 39.52% and 59.46% for Full.
W2.3 — Search-order robustness remains unclear. The released retriever consumes its 20-edge expansion budget in adjacency-list order, so changing the order of otherwise identical outgoing edges can change which paths are explored.
Questions for authors
Q1. Candidate construction and reference access (W1.1). What exact procedure produces the clean candidates used in Table 1, and which steps, if any, use reference mechanisms or intermediate answers? An evaluation on candidates generated from the query, known endpoints, and graph would help establish the contribution of Stage 1.
Q2. General selection and the saved candidate sets (W1.2). Are the saved at most three nodes the initial pools H0 or already selected sets H*? How do they relate to Figure 3? Please provide the original candidate pools, selection outputs, redundancy definition, coefficients, and executable selector. Which experiment tests the full selection rule against confidence-only selection when the pool exceeds the same Bridge budget?
Q3. Full configurations and Stage 3 attribution (W2.1–W2.2). Which candidate sets, confidence scores, query gates, controller parameters, and search settings produced Tables 1 and 4? Please explain the difference in Full performance and mean gate, provide the per-query records and generator for Table 4, and state how its configuration was selected. Do the constant, mean-matched, and shuffled-gate controls support the same conclusion when applied to the actual Table 1 Full configuration?
Q4. Search-order robustness (W2.3). What determines outgoing-edge order in the reported experiments? With models and predictions fixed, do the incremental gains persist across a small set of alternative orders or search budgets?
3 comments
Thank you for the detailed response and additional experiments. Revised Table 5 addresses my concern about aligning the gate controls with Table 1. However, I still have concerns as follows.
W1. Table 1 relies on supplied Clean candidates that already contain the intermediate answers, and their original construction remains incompletely documented.
The response confirms that these candidates were drawn from reference mechanisms, and Section 5.1 and Appendix B state that the intermediate answer is present in every test query's candidate pool. Their information source is therefore clear. Table 1 supplies reference-answer information to the retriever, so its large gains remain conditional on that access and do not establish the effectiveness of automatic candidate construction.
Appendix B also states that the historical candidate writer has not been authenticated. Please specify the original rule used to form each saved candidate set of at most three nodes, or identify which parts of that construction cannot now be verified. This matters for understanding what the main experiment actually evaluates.
W2. The reported DrugMechDB experiments do not establish the advantage of the complete three-stage combination under a matched comparison.
Section 4.2 describes multi-view candidate construction and specifies an automatic proposer using the query endpoints and graph. Sections 4.3–4.4 describe the complete greedy rule and query-specific influence control. My comparison of that design and the reported configurations is below. All table numbers refer to the revised manuscript.
| Source and configuration | Stage I / candidate source | Stage II | Stage III |
|---|---|---|---|
| Proposed method — Sections 4.2–4.4 | Multi-view candidate construction | Retrieval-aware Bridge reliability and the complete greedy rule | Query-specific influence control |
| Table 1 — Full (Section 5.2, p. 7) | Supplied reference-aligned Clean candidates | Learned confidences; all candidates retained without subset selection | Continuous gate |
| Appendix B — Full on fixed automatic proposal Top3 (pp. 15–16) | Automatic pools | Top3 selected by the proposer and assigned learned confidences; no complete greedy rule | Continuous gate |
| Table 4 — Complete greedy (Section 5.4, p. 9) | Automatic pools | Learned confidences and the reconstructed complete greedy rule | Fixed gate = 0.3 |
Table 4 is titled “Automatic-pool selection (K = 3, g = 0.3)”. The automatic-proposal Full result is reported separately in the Appendix B paragraph “Matched learned-path baseline”; it is not a row of Table 4.
Table 1's pools contain at most three distinct nodes, matching the selection budget, so this experiment also does not test nontrivial subset selection.
The automatic-proposal experiment in Section 5.3, also reported as Proposal Top3 in Table 4, does show a real benefit: guidance with every Bridge confidence set to 1 and g fixed at 0.3 improves PS/ER from 27.02/33.97% to 35.08/38.73%. This supports the usefulness of automatically proposed Bridges. However, the automatic-proposal Full result and the Complete greedy result use the two different configurations shown above. A matched evaluation establishing the advantage of automatic proposal + complete greedy selection + query-specific influence control over simpler alternatives remains missing from the evidence discussed here.
W3. The complete Stage-II selector and Stage-III control have not demonstrated stable incremental gains over the matched simpler alternatives.
- Stage II — Table 4, Complete greedy versus Confidence-only: utility improves by 0.00554, with a paired 95% CI of [−0.01420, 0.02451]. This does not establish a stable advantage of the complete selection rule.
- Stage III — Table 5, Full versus Validation-mean constant: utility improves by 0.00258, with a paired 95% CI of [−0.00072, 0.00861]. This likewise does not establish a stable incremental advantage of query-specific influence control. These controls use the actual Table 1 Full configuration.
These intervals do not prove that the components have zero effect; they leave their incremental advantage over these simpler controls uncertain.
I recognize the positive results for retrieval-aware Bridge reliability in Table 2 and the small combined gain from learned confidences and the continuous gate in the Clean setting: Table 1 Full exceeds Semantic+Fixed by 0.90 ER percentage points, with a 95% CI of [0.23, 1.69] (Section 5.2). Nevertheless, most of the main-table improvement is already obtained with fixed guidance from the supplied candidates, and the additional experiments do not convincingly establish the value of the complete proposed combination.
I find their methodological significance and the demonstrated benefit of the full framework insufficient for acceptance. I therefore maintain my recommendation of clear rejection.
Thank you for your continued engagement. We address W1 first, then W2 and W3 together: the framework's design and the evidence for its individual decisions.
W1. Clean construction and verification scope
Table 1 supplies reference-aligned node identities, including the intermediate answer, but no connecting path or relation sequence. Its gains are conditional on these cues; comparisons sharing them examine their use. Automatic construction and nontrivial subset selection are evaluated separately.
The executable Clean rule builds a global entity-type map in reference-file order, retaining the last label for each canonical node ID. For each query, traverse its mechanism's nodes in stored order, exclude known endpoints and deduplicate IDs. Insert the reference answer first, preferentially add unrepresented types, then fill in record order to three base nodes. Append the first eligible node of the answer's type if absent.
This reproduces all 1,354 TRAIN, 239 VAL and 248 TEST lists, including order, in both saved tables: 3,682 exact matches. Five TRAIN lists have four nodes; all VAL/TEST lists have at most three. The saved inputs are exactly reproducible; historical writer identity and execution traces remain unauthenticated. The automatic proposer uses the query, known endpoints and graph, without query-specific answers or reference membership.
W2-W3. Research questions, matched evidence and contribution
We distinguish two questions: what each retrieval decision does under controlled conditions, and whether the complete combination stably outperforms simpler alternatives. Our controlled design addresses the former; the latter is a stronger claim assessed separately. An unestablished overall advantage does not replace evaluation of the individual research questions.
The experiments have explicit, different purposes. Section 5.3 tests whether automatically proposed Bridges improve retrieval. Table 2 tests whether downstream usefulness is distinguishable from semantic similarity. Table 4 holds the automatic pool, confidence scores, K and guidance strength fixed to test the complete selection rule. Table 5 holds candidates and confidences fixed to examine query-specific control. Table 1 studies weighting/control under supplied Clean access. These configurations isolate decisions rather than treating every row called "Full" as the same complete pipeline. We do not add gains across them to infer a combined advantage.
The supported findings are substantive. As you recognize, automatic Top3 guidance improves PS/ER from 27.02/33.97% to 35.08/38.73%, without query-specific reference access. Table 2's Clean usefulness AUROC is .815 versus lexical .515; its matched single-Bridge evaluation raises ER from 38.19% to 39.92%. This distinguishes recovering a plausible candidate from identifying one useful to budgeted evidence retrieval. Table 1 Full exceeds Semantic+Fixed by .90 ER points, with a paired 95% CI [.23,1.69]; this is a combined weighting/control increment, with its scope stated explicitly.
Selection and control require more specific conclusions. Your cited utility intervals, complete greedy versus confidence-only +.00554 [-.01420,.02451] and Clean Full versus the validation-mean constant +.00258 [-.00072,.00861], do not establish stable independent superiority. We retain that qualification. Within the selector, however, the matched TypeBonus ablation gives +2.02 PS points [.40,4.02] and +.01588 utility [.00375,.03138]. These pointwise, unadjusted intervals support type complementarity in that configuration; they do not establish redundancy's independent benefit or replace the complete-rule comparison. Likewise, shuffled-gate controls and positive/negative cases characterize sensitivity to the query-strength correspondence, while strong constants delimit the independent gate claim.
The complete automatic matched evaluation also separates overall performance from these component findings. It does not establish stable overall superiority, and its query gate matches the validation-selected constant in per-query PS/ER/utility. We therefore make no claim that every proposed decision is independently necessary for the largest reported gain. That limitation is explicit; it does not remove the positive evidence for automatic Bridge guidance, retrieval-aware usefulness estimation, or the value of identifying where additional selection/control helps and where simpler choices suffice.
We ask that the paper be evaluated on its documented contribution: mechanism-evidence retrieval, the separation of availability, utility and influence, and controlled evidence of their roles and limits. Consistent full selector/gate superiority is a separate claim, not a substitute for assessing these findings.
We hope these clarifications address your concerns and make the contribution and its evidence clear. Thank you again for your time and consideration.
Thank you for recognizing our Stage-II usefulness evidence and verifying the main retrieval results. Our answers follow.
Q1 / W1.1 — Candidates and reference access
Table 1 uses supplied reference-aligned Clean Bridges drawn from reference mechanisms, including intermediate answers, to isolate subsequent retrieval decisions under controlled candidate access. They are not automatic-proposer outputs; our framework also supports automatic generation.
The proposer explores two forward rounds from the source, then two reverse rounds from the target, sharing a visited set. It excludes endpoints, groups candidates into molecular/cellular/system views by entity type, and deduplicates canonical IDs. A fixed rule combines lexical relevance, endpoint adjacency, anti-hub weighting and relation quality. Inputs are known endpoints/names, query intent and the fixed graph, excluding held-out intermediate answers and query-specific reference membership.
Before budget truncation, useful-candidate coverage rises from 40.32% for the strongest view to 61.29% for the union. Automatic Top3 with equal confidence and g=0.3 improves PS/ER over Graph Only by 8.06/4.76 percentage points (paired 95% CIs [4.40,12.11]/[2.88,6.84]). Sections 4.2 and 5.1–5.3 clarify both settings.
Q2 / W1.2 — Saved sets and greedy selection
Saved ≤3-node sets are supplied retriever inputs, not complete-greedy outputs. Table 1 Semantic+Fixed versus Utility+Fixed tests confidence weighting; Figure 3 describes larger automatic pools.
Parts of the original selector implementation could not be recovered. We reimplemented the stated rule with explicit operational definitions and evaluated it on automatic pools; this is not recovery of the historical implementation.
Starting empty, greedy maximizes Gain=c−μRed+βTypeBonus until K=3 or pool exhaustion. Red is maximum one-hop undirected-neighbor Jaccard with selected nodes (zero when empty); TypeBonus indicates a new entity type; ties use ascending node ID. Validation fixes C=0.1, μ=0.5, β=0.1. Both strategies share confidence scores, K=3, g=0.3 and search budgets. Of 248 pools, 239 exceed K; greedy changes 101 subsets versus confidence-only.
Both means improve numerically; utility differs by +0.00554 (paired 95% CI [−0.01420,0.02451]), without establishing stable superiority. Complete-selector results (§5.4, Table 4, Appendix B) use this implementation on previously inspected data. Its code, automatic pools, outputs and traces are released in the anonymous repository.
Q3 / W2.1–W2.2 — Configurations and attribution
Table 1 evaluates the overall framework; original Table 4 intended to diagnose guidance control, but its Full configuration (mean gate 0.10917) was not aligned with Table 1. Revised Table 5 corrects this.
Controls now share Table 1 Full's Clean candidates, confidences, query features, frozen predictions and search. All 248 Full outcomes match: PS/ER=39.52/59.46%, mean g=0.36035.
The gate is g=0.5√p_rel. A 42-setting grid over five validation candidate conditions selected g_max=0.5, γ=0.5 using utility, with nearest-0.05 cached gates for selection and continuous test gates. Clean validation utility selects constant 0.45; validation gates give mean constant 0.35877. Five hundred global shuffles preserve the test-gate distribution. Search fixes beam=2, hops≤8, nodes≤14, edge considerations≤20 and original adjacency order.
Full has highest mean utility, without stable independent gate superiority. Full-minus-validation-mean paired 95% CIs are ER [−0.2490,0.0000] points and utility [−0.00072,0.00861].
For W2.2, Bridges supply most improvement; learned components refine their use. Full-minus-Semantic+Fixed ER is +0.90 points [0.23,1.69], a combined weighting/control increment, not gate alone. Results are in §5.2/5.5, Table 5 and Appendix B; code, generator, validation configurations, per-query records and assignments are in the anonymous repository.
Q4 / W2.3 — Search-order robustness
Outgoing edges follow graph insertion order; duplicate (neighbor,relation) pairs retain first occurrence. The 20-edge budget follows this order. With models, predictions and budgets fixed, mean gains remain positive across original, sorted and ten seeded random orders:
We retain the original order for main results (§5.5, Appendix B).
Thank you again for your feedback.