- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces GenBench, a genomic knowledge graph (about 90k nodes) with 5,084 auto-generated multi-hop, cross-modal QA items, and CoRA, a Qwen3-8B agent trained with SFT and GRPO plus three add-on mechanisms.
Critical strengths
The paper is unusually candid about its limits, reporting bugs, weak signals and small ablation effects instead of hiding them.
Critical weaknesses
The main comparison is confounded, since a trained model with graph access is compared against zero-shot models without any tools; there’s no baseline that uses the same graph tools without training, or a standard RL search agent, so neither benchmark difficulty nor the method’s contribution is really isolated. GRAD, ROC and CoCG together add at most about 1.5 points over plain GRPO on a 200-item single run, the authors’ own diagnostic shows the ROC signal is slightly mis-signed, and CoCG’s benefit is unverified, so the core method claims aren’t supported. Benchmark quality is thin: items come from graph templates, the expert audit covers only 45 non-random items with one annotator, and about one in six answers was judged wrong; multiple-choice types also sit near ceiling and look more like graph lookups than reasoning. Several open-ended tasks, like “name a variant” or “shortest path”, can have many valid answers but are scored against a single gold string. The paper also reads unfinished, with the importance ratio always equal to one, multiple late bug fixes, and leftover notes in the references.
Questions for authors
What share of test items can be answered with a single tool call on the question’s own endpoints, and how does accuracy look once those items are removed?
Response to Reviewer 3Y1D — Ablations, Benchmark Audit, Multi-Hop Retrieval, and Evaluation
We thank the reviewer for the detailed assessment. We addressed the main concerns through full-test-set experiments, implementation checks, and an additional benchmark audit, documented in the revised manuscript.
W1–W2 — Controls and ablation significance
On the 1,019-item released test set, CoRA achieves 0.8626, compared with 0.8057–0.8126 for vanilla GRPO and three component ablations: +5.0–5.7 percentage points, with paired exact McNemar p < 0.0001 in all four comparisons (Appendix F.1). Additional controls yield 0.1668 for an untrained agent with graph/tools, 0.4868 for no-tool SFT, and 0.7184 for tool-enabled SFT without RL (Appendix F.2, F.6–F.7). The single-seed design limits mechanism-specific causal attribution.
W3–W4 — ROC and CoCG diagnostics
The ROC credit diagnostic shows a small negative four-view separation (−0.0135). CoCG raises top-20 gold-edge coverage from 50.0% to 53.8%, but does not consistently improve mean rank (72.6 to 74.0). We distinguish the full framework’s measured ablation advantage from claims of independent credit calibration or improved graph-confidence reliability (Appendix D.3, F.4).
W5 — Benchmark annotation quality
We conducted a second 200-item stratified audit of the filtered release, with two domain experts and third-expert adjudication. 185/200 answers (92.5%) were judged correct, 173/200 reasoning chains (86.5%) fully valid, and 20/200 (10.0%) partially valid. Answer-correctness agreement was 95.5% (Cohen’s κ = 0.552). Separate model-assisted verification confirmed six errors (3.0%), including five already identified by adjudication and one additional error; 15 other cases remain unresolved. These categories are not additive (Appendix F.11).
W6/Q1 — Tool lookup versus multi-hop reasoning
One QUERY-KG call reaches the gold path in 1,017/1,019 items when supplied with oracle-known gold endpoints; this does not demonstrate that the model can identify an unknown answer endpoint. Restricting QUERY-KG to one hop reduces accuracy from 0.8626 to 0.7076, including path traversal from 0.9600 to 0/150. Thus, the measured multi-hop retrieval capability belongs to the tool-supported system, not necessarily independent graph traversal by the language model (Appendix F.3).
For multiple-choice questions, answer-position and question-text-only baselines perform near chance; their largest observed advantage over chance is +0.065. This rules out the tested simple priors, not all possible shortcuts (Appendix F.9). Stronger graph-dependence evidence comes from open-ended results: 397/445 correct for CoRA versus 28/445 for no-tool SFT (Appendix F.6–F.8).
W7 — Acceptable answers and tied paths
Graph-valid alternative-answer lists cover 44/93 disease-reasoning and 196/202 counterfactual items. For the same CoRA checkpoint, disease accuracy changes from approximately 0.43 to 0.83, and counterfactual from 0.04 to 0.87, under single-reference versus acceptable-answer scoring; this is a scoring difference, not a training gain. We also identified tied shortest paths in 72/150 path-traversal items; these additional alternatives are not yet included in reported accuracy (estimated overall effect ≈0.003; Appendix F.10).
W8 — Implementation reliability
We consolidated implementation fixes in Appendix D, corrected the evaluation decoder and KL-reference handling, and documented the single-inner-epoch importance-ratio setting. A CPU dry-run confirmed clipping activates after a parameter update; the reported runs retain the documented single-inner-epoch configuration. All revised trained-system evaluations use the corrected decoder.
We appreciate the reviewer’s suggestions, which have materially strengthened the revised evaluation and its presentation.