Why Sample What You Can Enumerate? Exact Policy Optimization for Genomic Tool Selection

Reinforcement learning over a frozen reasoner has become a common recipe for teaching a policy which tools to invoke. We show that this recipe becomes structurally mismatched in specialist scientific settings where the complete tool-subset space is enumerable. There, a few recurring capabilities cover the domain, so the subset space is combinatorial yet small enough to enumerate, and GRPO still estimates an action expectation from a handful of sampled rollouts. Worse, the approximation degrades as the policy concentrates: it resamples its preferred subsets, sampled rewards collide, and the group-normalized advantage vanishes. On genomic reasoning the closed-form fraction of questions yielding no reward signal rises from $0.2\%$ under a uniform reference policy to $20.8\%$ after GRPO training. As a remedy, we introduce FGPO (Full-Group Policy Optimization), which (1) scores every tool subset and optimizes the exact action expectation, and (2) precomputes the reward of each question–subset pair into an exhaustive table, removing reasoner calls from training. Across five frozen reasoners and three genomic benchmarks, FGPO outperforms GRPO in all $15$ settings by $6.75$ points on average and up to $14.20$, while a standard on-demand GRPO schedule would require $2.4\times$ as many frozen-reasoner reward evaluations and, on GenomeQA, FGPO cuts invoked tools per question from $2.36$ to $1.40$. With the same table, it beats supervised routing and matches or beats reward regression in every setting.