acceptodds
Under review as a conference paper at ICLR 2027

GENBENCH: A CROSS-MODAL MULTI-HOP GE- NOMIC REASONING BENCHMARK WITH CONFIDENCE- COEVOLVING AGENTIC LEARNING

Abstract

Large language models answer genomics questions fluently, but their performance degrades sharply when answering requires chaining multiple pieces of evidence rather than recalling an isolated fact. We introduce GenBench, a 90,379-node, 236,131-edge genomic knowledge graph benchmark built around 210 manually curated seed genes and 5,084 automatically generated, multi-hop, cross-modal question–answer (QA) items, quality-checked by an automated verification pass and audited on a sample by domain experts. On this benchmark, zero-shot Qwen3-8B achieves only 0.2304 accuracy, while GPT-4o and Claude Opus-5 remain below 0.32. Specialized biomedical baselines, including BioReason, Intern-S1-mini, and Evolla, achieve overall accuracies of 0.1676, 0.2807, and 0.2950, respectively, with BioReason dropping to 0.027 on the subset requiring sequence-level evidence. We develop an evidence-aware agent named CoRA (Co-evolving Retrieval Agent) that acquires cross-modal evidence from the graph during reasoning and jointly learns retrieval, trajectory-level credit assignment, and graph confidence through four coupled mechanisms: GRAD (Gated Reward Advantage Distillation), ocbook (open-/closed-book), ROC (Rescored On-policy Credit), and CoCG (Co-evolving Confidence Graph). With supervised warm-start training and graph-based tool access, Qwen3-8B reaches 0.5329 accuracy, increasing to 0.5948 with our full GRPO training procedure, a 158% relative improvement over the zero-shot baseline. The gain comes from evidence access rather than scale, which shows that GenBench resists parametric-only solutions and rewards explicit, confidence-aware retrieval for compositional genomic reasoning. We further identify a substantial multiple-choice/open-ended gap and provide ablations and implementation diagnostics clarifying the training framework's behavior. We release the graph, QA items with complete reasoning chains, and training code.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
genomic reasoning, knowledge graph, multi-hop question answering, cross-modal reasoning, agentic reinforcement learning, credit assignment, LLM agents, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3Y1DOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces GenBench, a genomic knowledge graph (about 90k nodes) with 5,084 auto-generated multi-hop, cross-modal QA items, and CoRA, a Qwen3-8B agent trained with SFT and GRPO plus three add-on mechanisms.

Critical strengths

The paper is unusually candid about its limits, reporting bugs, weak signals and small ablation effects instead of hiding them.

Critical weaknesses

The main comparison is confounded, since a trained model with graph access is compared against zero-shot models without any tools; there’s no baseline that uses the same graph tools without training, or a standard RL search agent, so neither benchmark difficulty nor the method’s contribution is really isolated. GRAD, ROC and CoCG together add at most about 1.5 points over plain GRPO on a 200-item single run, the authors’ own diagnostic shows the ROC signal is slightly mis-signed, and CoCG’s benefit is unverified, so the core method claims aren’t supported. Benchmark quality is thin: items come from graph templates, the expert audit covers only 45 non-random items with one annotator, and about one in six answers was judged wrong; multiple-choice types also sit near ceiling and look more like graph lookups than reasoning. Several open-ended tasks, like “name a variant” or “shortest path”, can have many valid answers but are scored against a single gold string. The paper also reads unfinished, with the importance ratio always equal to one, multiple late bug fixes, and leftover notes in the references.

Questions for authors

What share of test items can be answered with a single tool call on the question’s own endpoints, and how does accuracy look once those items are removed?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 3Y1D — Ablations, Benchmark Audit, Multi-Hop Retrieval, and Evaluation

We thank the reviewer for the detailed assessment. We addressed the main concerns through full-test-set experiments, implementation checks, and an additional benchmark audit, documented in the revised manuscript.

W1–W2 — Controls and ablation significance

On the 1,019-item released test set, CoRA achieves 0.8626, compared with 0.8057–0.8126 for vanilla GRPO and three component ablations: +5.0–5.7 percentage points, with paired exact McNemar p < 0.0001 in all four comparisons (Appendix F.1). Additional controls yield 0.1668 for an untrained agent with graph/tools, 0.4868 for no-tool SFT, and 0.7184 for tool-enabled SFT without RL (Appendix F.2, F.6–F.7). The single-seed design limits mechanism-specific causal attribution.

W3–W4 — ROC and CoCG diagnostics

The ROC credit diagnostic shows a small negative four-view separation (−0.0135). CoCG raises top-20 gold-edge coverage from 50.0% to 53.8%, but does not consistently improve mean rank (72.6 to 74.0). We distinguish the full framework’s measured ablation advantage from claims of independent credit calibration or improved graph-confidence reliability (Appendix D.3, F.4).

W5 — Benchmark annotation quality

We conducted a second 200-item stratified audit of the filtered release, with two domain experts and third-expert adjudication. 185/200 answers (92.5%) were judged correct, 173/200 reasoning chains (86.5%) fully valid, and 20/200 (10.0%) partially valid. Answer-correctness agreement was 95.5% (Cohen’s κ = 0.552). Separate model-assisted verification confirmed six errors (3.0%), including five already identified by adjudication and one additional error; 15 other cases remain unresolved. These categories are not additive (Appendix F.11).

W6/Q1 — Tool lookup versus multi-hop reasoning

One QUERY-KG call reaches the gold path in 1,017/1,019 items when supplied with oracle-known gold endpoints; this does not demonstrate that the model can identify an unknown answer endpoint. Restricting QUERY-KG to one hop reduces accuracy from 0.8626 to 0.7076, including path traversal from 0.9600 to 0/150. Thus, the measured multi-hop retrieval capability belongs to the tool-supported system, not necessarily independent graph traversal by the language model (Appendix F.3).

For multiple-choice questions, answer-position and question-text-only baselines perform near chance; their largest observed advantage over chance is +0.065. This rules out the tested simple priors, not all possible shortcuts (Appendix F.9). Stronger graph-dependence evidence comes from open-ended results: 397/445 correct for CoRA versus 28/445 for no-tool SFT (Appendix F.6–F.8).

W7 — Acceptable answers and tied paths

Graph-valid alternative-answer lists cover 44/93 disease-reasoning and 196/202 counterfactual items. For the same CoRA checkpoint, disease accuracy changes from approximately 0.43 to 0.83, and counterfactual from 0.04 to 0.87, under single-reference versus acceptable-answer scoring; this is a scoring difference, not a training gain. We also identified tied shortest paths in 72/150 path-traversal items; these additional alternatives are not yet included in reported accuracy (estimated overall effect ≈0.003; Appendix F.10).

W8 — Implementation reliability

We consolidated implementation fixes in Appendix D, corrected the evaluation decoder and KL-reference handling, and documented the single-inner-epoch importance-ratio setting. A CPU dry-run confirmed clipping activates after a parameter update; the reported runs retain the documented single-inner-epoch configuration. All revised trained-system evaluations use the corrected decoder.

We appreciate the reviewer’s suggestions, which have materially strengthened the revised evaluation and its presentation.

Reviewer_H6UbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces GenBench, a knowledge-graph-based benchmark for genomic reasoning, and CoRA, an agentic framework that combines graph retrieval, reinforcement learning, and confidence updates. The authors report improved question-answering performance, but the evidence supporting the proposed mechanisms remains limited.

Critical strengths
  1. Relevant research problem. The paper addresses evidence-grounded genomic reasoning and provides a benchmark with explicit reasoning paths.

  2. Promising system integration. The combination of graph retrieval and agentic learning offers a potentially useful direction for biomedical question answering.

Critical weaknesses
  1. Insufficient experimental controls. The benchmark questions are constructed from a knowledge graph that the proposed agent can access, while the main baselines lack comparable graph access and training. Consequently, the reported improvements cannot be clearly attributed to CoRA's proposed mechanisms.

  2. Unconvincing validation of co-evolution. The contribution of confidence updates is marginal in the reported ablations, and the reliability of the credit-assignment mechanism remains unclear. More importantly, the paper does not establish whether the updated graph confidences better reflect biological evidence or whether the method supports generalizable cross-modal reasoning.

Questions for authors
  1. Can the authors provide comparisons with baselines using the same knowledge graph, retrieval tools, and training budget?

  2. What independent evidence demonstrates that the confidence updates improve knowledge reliability and generalize beyond the benchmark constructed from the same graph?

1 comment
AuthorsOpenReview ↗

Response to Reviewer H6Ub — Baseline Controls and Graph-Confidence Validation

We thank the reviewer for highlighting the need for stronger controls and evidence regarding graph-confidence updates. We added targeted evaluations in Appendix F.

W1/Q1 — Matched graph and tool-access controls

An untrained Qwen3-8B agent with the same graph and tools achieves 0.1668; 61.3% of items contain unparseable actions, showing that access alone does not ensure effective tool use (Appendix F.2). Additional controls achieve 0.4868 for no-tool SFT, 0.7184 for tool-enabled SFT without RL, and 0.8626 for full CoRA (Appendix F.6–F.7). These experiments distinguish tool availability from learned tool utilization; they are not a compute-matched comparison against a separately trained RL search method.

W2 — Contribution of co-evolution

Removing CoCG reduces accuracy from 0.8626 to 0.8116 (paired exact McNemar p < 0.0001) on the 1,019-item test set (Appendix F.1). This supports CoCG’s role within the integrated framework, without implying that the confidence updates independently improve biological calibration.

Q2 — Independent confidence evidence

We added confidence-ranking and graph-dependence diagnostics (Appendix F.4–F.5). On disease reasoning, top-20 gold-edge coverage changes from 50.0% on the initial graph to 53.8% on the evolved graph; however, mean rank changes from 72.6 to 74.0, so there is no consistent ranking improvement. We therefore do not present this as independent confidence-calibration evidence.

Randomly removing 20% of graph edges reduces CoRA accuracy from 0.8626 to 0.7556, with larger declines on graph-dependent open-ended tasks. This highlights reliance on graph evidence, while independent-graph generalization remains an important direction for future validation.

We appreciate the reviewer’s suggestions, which motivated these additional controls and a more precise statement of the supported claims.

Reviewer_HmjROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces GenBench and CORA. GenBench is a benchmark constructed from 18 public biology database. The authors constructed a KG with more than 90 thousand nodes and 236 thousand edges, with each edge comes with a confidence score, and 7 modalities of payload. 5K QA pares are generated with the k-hop reasoning paths in the KG. CoRA introduces several mechanisms into a GRPO rollout batch, ocbook makes the agent decide itself to retrieve from the KG or answer directly, GRAD adds a weight to the reward based on path confidence, ROC gets four different perspectives of confidence and CoCG evolves the confidence score on the edges. Experiments show that parametric knowledge is insufficient on GenBench: GPT40 gets around 0.32 and Claude Opus gets around 0.25. The biology specific models get 0.17-0.3. RAG prompting baseline reaches 0.4090. With graph access and training, Qwen3-8B improves to 0.5329 after ocbook SFT and 0.5948 after full GRPO. Authors included ablation studies on a 200-item subset, which shows that vanilla GRPO reaches 0.6150 and removing GRAD, ROC, or CoCG individually costs at most 0.015, so most of the gain comes from SFT and graph access rather than the proposed mechanisms.

Critical strengths

The benchmark is carefully constructed. GenBench introduces a large heterogeneous KG with edge confidence that directly comes from the data source, and 7 modalities on the relation. The dataset comes with complete reasoning chains that can be directly used as SFT, rather than only evaluation rubrics. This will benefit further research in search agent, knowledge retrieval/generation, and KG exploration/reasoning.

Critical weaknesses

My major concern is the ablation does not support the paper's central methodological claim. Table 2 shows that vanilla GRPO reaches 0.615 without the four mechanisms introduced (GRAD, ROC, CoCG, ocbook), whereas the full CoRA reached 0.63. This is a 3-item difference on a 200-item subset from a single run with no seeds or confidence intervals. Removing any single mechanism costs between 0.005 and 0.015, which is within noise for 200 items.

Questions for authors
  1. What is the difference between the training objective of vanilla GRPO versus Full? Eq. 5 is the standard GRPO group-relative advantage, if GRAD’s only addition is the path-confidence gating in Eq. 4, does the “no GRAD” configuration train on raw correctness reward with the same normalization?

  2. why the ablation uses a 200-item subset rather than the full 1020-item test set, and can you try multiple seeds to see if the reported delta changes?

  3. Can you test SFT on the same QA items without any graph tools or modality embeddings? This will help us understand if the gain comes from evidence access or learning the template structure.

1 comment
AuthorsOpenReview ↗

Response to Reviewer HmjR — Full-Scale Ablations and SFT Controls

We thank the reviewer for the constructive suggestions. We strengthened the evaluation through full-scale ablations and additional training controls.

1. Full-scale ablation and statistical significance

All five configurations were evaluated on the released 1,019-item test set using the corrected decoder (Appendix F.1):

ConfigurationAccuracy
Full CoRA0.8626
w/o GRAD0.8126
w/o CoCG0.8116
w/o ROC0.8077
Vanilla GRPO0.8057

CoRA exceeds every ablation by 5.0–5.7 percentage points (paired exact McNemar p < 0.0001 for all four comparisons); gene-cluster bootstrap intervals support these differences. Training lengths and implementation settings are matched. As the runs are single-seed, the results support the combined framework without establishing independent causal contributions for each component.

2. GRAD ablation clarification

Removing GRAD disables the gold-path reward, verification-depth credit gating, and discovery bonus, while retaining ROC, CoCG, and group-relative normalization. Vanilla GRPO removes the additional mechanisms while preserving group-relative normalization (Appendix F.1; Appendix D).

3. SFT without graph tools or modality embeddings

We trained a no-tool SFT control on the same QA items, without tool interactions or modality embeddings (Appendix F.6). It achieves 0.4868, versus 0.8626 for CoRA. The difference is pronounced on open-ended tasks:

TaskNo-tool SFTCoRA
Disease reasoning0.00000.8280
Counterfactual0.08420.8713
Path traversal0.07330.9600

Tool-enabled SFT without RL achieves 0.7184 (Appendix F.7); disabling tools at CoRA evaluation reduces accuracy to 0.5201 (Appendix F.8). These controls support the value of trained graph access, while the no-tool comparison also differs in training objective and the tool-disabled evaluation changes the inference distribution.

We appreciate the reviewer’s feedback, which helped strengthen the experimental evidence and clarify the ablation design.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.