acceptodds
Under review as a conference paper at ICLR 2027

Prediction Targets and Vocabulary Coverage in Bacterial Pangenome Models

Abstract

Genome language models over bacterial pangenomes predict each gene as one of a fixed set of protein families; we compare nine prediction targets for such models under a matched corpus, backbone, masking scheme, token budget and evaluation. Contrary to our pre-registered primary hypothesis, a head tied to a frozen protein encoder's centroids has higher held-out cross-entropy than a free classifier, by 0.225 nats. The gap depends on distribution shift. It is 0.435 nats for unseen species of a seen genus and indistinguishable from zero at unseen phylum, where resampling phyla rather than species widens the bound to 0.051 nats and the species-level reversal seen with a 302M-parameter backbone does not survive. Part of the gap reflects head size: at matched trainable parameters, the tied head is ahead. The target further sets what a model can represent. Out-of-vocabulary genes are 21.25% of held-out gene mass, and a closed-vocabulary head cannot emit their labels at all; a tied head can, but its exact-family accuracy on them is near zero and its net advantage depends on how finely the vocabulary is extended. An auxiliary target predicting cross-taxon gene persistence works on either head at small cost. Finally, frozen-probe conclusions under the widely used protocol depend on the layer probed: at the final layer no target we tested beats a protein-only baseline, while at the first block every arm's mean does on essential genes.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
genomic language models, pretraining objectives, distribution shift, open-vocabulary prediction, protein language models, evaluation methodology, bacterial pangenomics, self-supervised learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dEeYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how the choice of prediction target affects the baterial pangenome language models. The authors compare nine variants under a controlled set-up with matched corpus, backbone, masking scheme and token budget. The main comparison is between a conventional free 50k way classifier and a metric head tied to frozen esmc embeddings and protein family centroids. the paper also considers parameter-matched free classifiers, a permitted geometry control, a continuous regression target, and geometry based soft target. The main result is that the metric head perform worse than the full free classifier on the standard closed vocabulary task by 0.225 ants, although this gap decreases with increasing taxonomic distance and becomes statistically insignificant at unseen phylum. Although at matched trainable parameter count, the metric head significantly outperforms the free classifier, illustrating that the pretrained protein space geometry provide a useful inductive base under a constrained head budget.

Critical strengths
  1. Strong experimental controls The paper holds the data, backbone, masking scheme, training budget and evaluation mask fixed while varying the prediction target, This isolates a question that is often conflated in comparisons between existing genome language models. The 9-arm setup is very well constructed, and include parameter-matched free classfiers and a permutated centroid control

  2. The parameter matched and permutation controls are very strong. The full free classifier has many more trainable parameters than the metric head, so the authors appropriately introduce a parameter matched free classifier. At approximately matched trainable parameter count, the metric head is 1.22 nats better, while permuting the label to centroid correspondence significantly degrades the performance. This is good evidence that the pertrained geometry itself contributes useful information.

  3. Taxonomic distribution shift is well treated The authors evaluate increasingly distant taxonomic bands from unseen species to unseen phylum. The finding that the metric head deficit shrinks significantly with more toxonomic distance is interesting to see.

Critical weaknesses
  1. Limited methodological novelty My main concern is technical novelty, as the authors explicitly it described it as an established metric classifier over frozen centroids and state that the contribution is the measurement and not the head. The genome level transformer architecture is also tsandard relative to prior work such as gLM and Bacformer. The novelty rests primarily on the controlled empirical comparisons. I am not yet fully convinced that the resulting empirical findings are sufficiently broad or surprising to match the scope of ICLR

  2. The open vocab capability is technically present but empirically very weak. A major motivation of the paper is that a closed 50k way vocab cannot emit unseen protein families. However, under the granularity matched extension, the metric head obtains only approximately 0.17% family accuracy on novel families, and authors make no claims of the practical utility. So this makes the main practical motivation for the tied head weaker than the framing of the paper suggested.

  3. Overall, the broader ML takeaway remains somewhat unclear The most general conclusions seems to be that semantic output geometry can improve parameter efficiency, that its effect depends on the distributional shift and the open vocab evaluation depends strongly on label space construction. These are interesting observations but it would be interesting if the paper articulates more clearly which findings should be expected to generalize beyond this particular bacterial pangenome setup?

Questions for authors
  1. Given that the metric head itself is an established approach, what do the authors view as the main generalizable machine-learning contribution of this work beyond the bacterial pangenome setting? In particular, which of the observed behaviors do the authors expect to hold for other structured output spaces built from pretrained embeddings?

  2. How should the open-vocabulary results be interpreted in practice given the very low exact-family accuracy on novel families? Given the 0.17% accuracy, are there downstream use cases where coarse localization in embedding space is also useful even when exact family identification is not successful?

  3. How much of the genome Transformer’s learned signal comes from gene-content composition rather than meaningful gene ordering? Since the corpus ordering is not genomic coordinate and the authors explicitly avoid synteny/neighborhood claims, it would be useful to see whether an order-invariant/set-based baseline would perform similarly.

Reviewer_LX88OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper compares nine output heads for bacterial genome language models under the same model, data, and training budget. In particular, it compares a standard classifier over a fixed set of protein families with a metric head that, in principle, can also predict new families. The standard classifier performs better, and the metric head reaches almost zero accuracy on new families.

Critical strengths

The authors are very transparent, reporting negative results and the errors found during the work.

Critical weaknesses
  1. The paper does not introduce any novelty or significant empirical evidence that justifies a submission to a mainly machine learning conference. The authors themselves state that the architecture is standard and that the metric head is an established technique (“Our contribution is the measurement, not the head”). However, the main measurements lead to negative or inconclusive results: the primary hypothesis fails (Section 4), the soft target does not resolve the gap (Section 6), and the open-vocabulary capability that motivates the paper reaches an exact-family accuracy below 1% on novel families, so that the authors themselves make “no claim of practical utility” (Section 5.1).

  2. I have concerns regarding the soundness of the evaluation pipeline. First, the order of genes in the corpus does not correspond to their positions in the genome (Appendix A), so the model only sees which genes co-occur, not their organization along the genome. Second, the main open-vocabulary comparison between the metric head and the free classifier changes sign depending on evaluation choices, such as the granularity of the vocabulary extension or the ranking rule (Section 5).

  3. The paper is extremely hard to read. ICLR is mainly a machine learning conference, and terms such as “pangenome,” “phylum, “contig”, “core/accessory genes”, or “protein family” are not typical of the field, yet they are used without being introduced. If the intent is to submit to a conference of this type, in my opinion, the authors should make the effort to clarify the setting for a different community. Moreover, the paper's main message is very unclear.

Questions for authors
  1. The primary hypothesis of the paper is not verified (Section 4), the soft target does not resolve the gap (Section 6), and the authors themselves make no claim of practical utility for the open-vocabulary capability (Section 5.1). Given these results, what is the main claim the authors want to demonstrate, and what concrete recommendation can practitioners draw from this work?

  2. The out-of-vocabulary genes are defined by the authors’ own frequency filter. Since the test species have, on average, only about four genomes (4727 genomes across 1088 species), for most of them, a gene is retained only if it is present in all species' genomes. The out-of-vocabulary set, therefore, mostly consists of ordinary genes removed by this filter, not genuinely new protein families. What is then the utility of predicting these genes, given that they are excluded by the authors’ own preprocessing?

  3. The paper does not report the number of genomes per species in each taxonomic band. Could the authors report it? Without this information, it is not possible to tell whether the increase of out-of-vocabulary genes with taxonomic distance (Table 1) is due to distribution shift or simply to the filter.

  4. Why did the authors not train on the genomic order, and would the main conclusions hold in that setting?

Reviewer_Nd9VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether the choice of prediction head matters for transformer-based bacterial pangenome models, which predict each gene as one of a fixed set of protein-family prototypes. Nine heads — a free 50k-way classifier, a metric head tied to frozen ESM-C centroids, a parameter-matched free head, a permuted-geometry control, a continuous regression target, a geometry-shaped soft target, and a gene-persistence auxiliary head — are trained under an identical corpus, backbone, masking scheme, token budget and evaluation mask. The metric head loses on closed-vocabulary cross-entropy (+0.225 nats); the gap shrinks with taxonomic distance until indistinguishable from zero at unseen phylum, while at matched trainable parameters the tied geometry is ahead by ~1.2 nats. Vocabulary extension lets the tied head emit labels for out-of-vocabulary genes (21.25% of held-out gene mass) but with near-zero exact-family accuracy, and the sign of its advantage depends on extension granularity.

Critical strengths
  1. The empirical discipline is exceptional: the analysis plan was preregistered with nineteen separately committed amendments and pre-specified interpretations for adverse outcomes (four occurred, including the failed primary hypothesis); every number is generated into a 1,130-entry registry, of which 1,053 reproduce from the clean release; and an eleven-defect audit records, for each defect, whether it was found before or after reporting.

  2. The paper does not merely disclose the corpus-ordering artifact — it quantifies it and converts it into guidance: the token order is not genomic order (3.26% of contigs monotone in the true coordinate), evaluation-time shuffling costs ~0.49 nats, and training on a within-contig shuffle improves both arms by roughly 0.09 nats (arm A: 7.059 → 6.973 at the 4B budget).

  3. Fragile results are interpreted conservatively: the 302M sign flip at unseen phylum is shown to be carried by a single clade (Omnitrophota, 80 of 117 species) and to vanish under phylum-unit resampling, and the authors explicitly decline to claim a reversal rather than report the flattering reading.

Critical weaknesses
  1. Several headline numbers rest on very few seeds and are reported only as "directions": the 302M comparisons use two seeds (arm C at 302M, one), the parameter-matched advantage rests on three seeds of a small head whose spread is 6× arm B's, the permuted control is a single seed, and arm E's three-seed result has the opposite sign of its earlier one-seed result (§E.2). The unseen-phylum conclusion is dominated by one phylum. The disclosure is honest, but the most cited quantitative claims are among the least statistically supported.
  2. The open-vocabulary evaluation structurally excludes its extreme cases: ~599k genes (4.78 points of held-out gene mass) on contigs emptied by the frequency filter contribute to the 21.25% out-of-vocabulary share but can supply no evaluation position, so the near-zero exact-family accuracy and the granularity-dependent sign reversal are measured only on the milder subset of out-of-vocabulary genes.
  3. The most positive result (§7, gene persistence) is scored only against ŷ = 0 with a frozen family prior; the stronger baseline the authors themselves identify (persistence in the nearest training clade) was not run, so the R² attributable to genome context is not isolated, and the head differences (0.005–0.011 nats) sit within seed spread.
Questions for authors

Since genes on fully filtered contigs (~599k, Appendix B) have no evaluation position, could the open-vocabulary conclusions be tested or bounded on this extreme subset, for instance by inserting these genes into matched retained contexts? Do the authors expect the near-zero exact-family accuracy to hold there?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.