acceptodds
Under review as a conference paper at ICLR 2027

Why Sample What You Can Enumerate? Exact Policy Optimization for Genomic Tool Selection

Abstract

Reinforcement learning over a frozen reasoner has become a common recipe for teaching a policy which tools to invoke. We show that this recipe becomes structurally mismatched in specialist scientific settings where the complete tool-subset space is enumerable. There, a few recurring capabilities cover the domain, so the subset space is combinatorial yet small enough to enumerate, and GRPO still estimates an action expectation from a handful of sampled rollouts. Worse, the approximation degrades as the policy concentrates: it resamples its preferred subsets, sampled rewards collide, and the group-normalized advantage vanishes. On genomic reasoning the closed-form fraction of questions yielding no reward signal rises from under a uniform reference policy to after GRPO training. As a remedy, we introduce FGPO (Full-Group Policy Optimization), which (1) scores every tool subset and optimizes the exact action expectation, and (2) precomputes the reward of each question–subset pair into an exhaustive table, removing reasoner calls from training. Across five frozen reasoners and three genomic benchmarks, FGPO outperforms GRPO in all settings by points on average and up to , while a standard on-demand GRPO schedule would require as many frozen-reasoner reward evaluations and, on GenomeQA, FGPO cuts invoked tools per question from to . With the same table, it beats supervised routing and matches or beats reward regression in every setting.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
reinforcement learning, tool selection, exact policy optimization, large language models, genomic reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kouhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies tool selection for a frozen LLM reasoner on multiple-choice genomics questions. A policy (Qwen2.5-7B with LoRA) chooses which of four sequence-analysis tools to run before the reasoner answers: composition, motif scan, splice-site scoring, and a kNN predictor over Nucleotide Transformer embeddings. The authors argue that GRPO is ill-suited when all 2⁴ = 16 tool subsets can be enumerated. As the policy concentrates, the sampled rollouts often all get the same reward, so the group-normalized advantage is zero (a "dead group"). The closed-form dead-group rate on GenomeQA rises from 0.2% under a uniform policy to 20.8% after GRPO training.

The authors propose FGPO: precompute the reasoner's reward for every question–subset pair once, then maximize the exact expected reward over all 16 subsets. The objective uses a length-normalized softmax over the candidate strings plus an entropy bonus. Across three benchmarks (GenomeQA plus two suites, GenBench-X and BM4) and five frozen reasoners, FGPO beats GRPO in all 15 settings, by 6.75 points on average. It also calls fewer tools (1.40 vs 2.36 per question on GenomeQA). The paper also covers cover: (a) a sweep over how many of the 16 subsets the objective sees per question; (b) an ablation of the training objective (cross-entropy, regression); (c) checkpoint and hyperparameter sweeps for the baselines.

Critical strengths
  1. Careful diagnosis and many controls. The paper gives a closed-form dead-group analysis and varies how many subsets the objective sees, with accuracy rising as it sees more. All arms are also scored under the same inference rule, the objective is ablated, the baselines get checkpoint sweeps, and differences are tested with question-paired McNemar tests. This makes the paper's own measurements easy to audit.
  2. Consistent margins over GRPO with fewer tool calls. FGPO beats GRPO in all 15 benchmark–reasoner settings, including four reasoners the policy was never trained with, while using fewer tools and 3.5–4.1× fewer input tokens than calling all tools.
Critical weaknesses
  1. The learned routing adds little over a fixed choice, and the contribution is largely known. FGPO calls the kNN tool (T4) on 86–96% of questions and never calls T1. App. C.3 states that FGPO only "tracks" the best fixed subset, chosen using test labels, with a gap that goes both ways, and that baseline isn't in Table 2. A dev-selected "always use the best fixed subset" baseline, e.g., always T2+T4 or always T4, would likely match FGPO. The method claims a query-dependent policy, so this undercuts the main claim.

    Methodologically, maximizing the exact expected reward over a small known action set with a full reward table is a full-information contextual bandit, close to cost-sensitive classification and to expected policy gradients (Ciosek & Whiteson). A plain regression on the reward table ties FGPO on BM4. That GRPO underperforms in this regime is expected.

  2. The observations from baselines and benchmarks are not convincing.

    • Every baseline that could compete collapses to the no-tool floor at every checkpoint: DPO, GRPO trained on Eq. 1, and DAPO trained on Eq. 1, all within 0.4 points of No-Tool. Yet FGPO, on the same reward, calls T4 on 86% of questions. DAPO's dynamic sampling exists precisely to discard zero-advantage groups, so its collapse points to an implementation or reward problem rather than an inherent limit of sampling.
    • The reported GRPO uses a different reward from FGPO, and it scores below simply calling all tools.
    • GenBench-X and BM4 are not established benchmarks. The authors built them from Genomic Benchmarks, DeepSTARR and other sources. Label meanings were inferred heuristically, and two turned out reversed. Sources were assigned to task slots by hand, e.g., DeepSTARR enhancers filed as "TF motif".
    • T4 is a kNN classifier built on labelled training splits of the same sources, so "tool selection" largely reduces to "consult the classifier".
    • There is one training run per arm, with no seed variance.
    • The "2.4× fewer reasoner calls" figure is bookkeeping, not a measured saving: a cached GRPO needs at most the same 32,032 unique calls.
Questions for authors
  1. Most task families here (promoter/enhancer, splice site, histone marks, taxonomy) are standard sequence-classification tasks. On these, fine-tuned genomic models (e.g., DNABERT-2, Nucleotide Transformer, HyenaDNA, Evo) are the usual baselines, and the multiple-choice format can be handled by scoring each option's sequence. How does the FGPO pipeline compare with (a) the T4 kNN classifier used directly as the answer, with no LLM, and (b) a fine-tuned genomic classifier on the same splits? If these match or exceed the pipeline, what does an LLM with learned tool selection add for these tasks?
  2. How does FGPO compare in Table 2 with a dev-selected best fixed subset and with "always T4", on all 15 settings?
  3. Why does DAPO, whose dynamic sampling removes zero-advantage groups, still collapse to no tools at every checkpoint? Did you try GRPO with G=16, sampling without replacement, or the same entropy bonus FGPO uses?
  4. How were GenBench-X and BM4 built and labelled, how were sources assigned to task slots, and how was label correctness verified?
  5. Can you report results over several training seeds for FGPO and GRPO?
Reviewer_SBuAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a Full-Group Policy Optimization (FGPO) method for tool selection in specialist scientific reasoning. Unlike GRPO, which estimates action expectations from a limited number of sampled rollouts and suffers from diminishing reward signals as policies concentrate, FGPO exhaustively evaluates all possible tool subsets and precomputes question–subset rewards, enabling exact policy optimization while eliminating reasoner calls during training. Experiments across five frozen reasoners and three genomic benchmarks show that FGPO achieves consistently better performance than baselines.

Critical strengths
  1. The paper is well-organized, with a clear and logical structure.
  2. The proposed FGPO is well-motivated for genomic tool selection when the tool subset space is enumerable.
  3. The reported experimental results significantly outperform baseline methods with an enumerable tool set.
Critical weaknesses
  1. The proposed method relies on exhaustively enumerating tool subsets, which is practical when the number of available tools is relatively small. However, the number of possible subsets grows exponentially with the number of tools, potentially limiting the applicability of FGPO in more complex settings. It would be better to analyze its scalability with substantially larger tool sets, such as those containing more than 100 tools.
  2. A more comprehensive analysis of the overall computational efficiency would be valuable.
  3. All experiments are conducted on genomic question-answering benchmarks, leaving the generalizability of FGPO to other scientific or specialized domains insufficiently explored. It would be valuable to evaluate the method on additional domains, such as biomedical research, chemistry, or scientific literature analysis, to determine whether its advantages extend beyond genomic reasoning and to identify the conditions under which exhaustive tool-subset optimization is most effective.
Questions for authors

Please refer to the Critical Weaknesses.

Reviewer_trw8OpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at RL-trained tool-subset selection for a frozen LLM reasoner on genomic multiple-choice QA, using four tools and therefore 16 possible subsets. Its main argument is that when the action space can be fully listed, GRPO's sampled, group-normalized advantages are the wrong tool. As the policy concentrates, the sampled rewards in a group tend to be identical, so the advantage is zero and that group teaches nothing,

Critical strengths

The question is clearly posed and practical: why sample an expectation you can compute exactly? The dead-group diagnostic is computed in closed form, backed by Proposition 1, and also measured live during training.

Critical weaknesses

With a precomputed reward table and 16 actions, FGPO is supervised learning with full feedback: a softmax over candidates trained on expected reward, close to cost-sensitive classification. Expected or all-action policy gradients are well known. The "RL vs. RL" framing makes GRPO look like the natural competitor. Stronger full-information baselines are missing: GRPO/RLOO with G≥16 or sampling without replacement, and a lightweight classifier on sequence embeddings.

Questions for authors
  1. How does GRPO or RLOO do with G=16, or with sampling without replacement, on the same reward table?
  2. Did you try a lower KL coefficient or a different temperature?
  3. What is the dead-group rate of GRPO at initialization under the Eq. 1 reward classes?
  4. What is FGPO's free-generation accuracy if out-of-range outputs are scored as invalid rather than repaired by the parser?
  5. Can you report deviation over at least 3 seeds for FGPO, GRPO and reward regression?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.