acceptodds
Under review as a conference paper at ICLR 2027

TaxoSet: Open-Reference Taxonomic Assignment of DNA Barcodes

Abstract

A classifier can only return a name from its label space, and in DNA barcoding that space is the sequence reference, not the whole taxonomy. Our four curated corpora carry a barcode for of the species it names. For a query from any of the other , a method that can only return reference names has an accuracy of exactly zero, whatever the model. We introduce open-reference taxonomic assignment. The reference supplies the evidence, a query's nearest sequences and their lineages. The taxonomic hierarchy, 2.9 million taxa at all ranks, supplies the candidate names, sequenced or not. When the evidence locates a branch but cannot separate its unsequenced members, choosing one would invent information, so the method returns a set of names, whose coverage we never report without its size. TaxoSet implements this with a 0.34M-parameter network that learns how to decide, not what the taxa are, so it applies to names it never saw in training. Removing whole clades from the reference before retrieval makes novelty a label rather than an estimate. On ITS queries whose genus is absent, the returned set holds the true genus of the time at a median of 11 names. At species rank it falls to , where the limit is detecting novelty rather than the set itself. On a third-party benchmark whose novelty is real rather than simulated, TaxoSet reaches on ITS, where no published method returns the absent genus at all.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
open-world learning, DNA barcoding, hierarchical classification, conformal prediction, set-valued classification, open-set recognition, taxonomic classification, computational biology

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Qw7uOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes TaxoSet, an algorithm to retrieve taxa from DNA barcodes when only a handful of taxa are sequences. It proposes a neural network system to decide when the given DNA barcode corresponds to a mapped taxon, whether it corresponds to one that is not included in the references or if it is only related (children) to one of them. In the latter case the algorithm returns a set of all plausible children that could correspond to the given barcode (when no information allows to differentiate between them). The algorithm reaches fairly good classifications capabilities on various markers.

Critical strengths
  • the paper addresses an interesting topic which could be particularly useful in practice in biology
  • it explains in depth through various experiments and ablations how the method works, its strengths and its weaknesses
Critical weaknesses
  • the paper does not clearly describe the training process (no optimization curve, how hyperparameters were chosen, ...). This does not give a clear view of how the core of the algorithm is trained, what are the challenges of training and whether the results found in the paper could actually be improved even further with training adjustments. These can usually be crucial information for future replications and if possible future iterations on this algorithm.
Questions for authors
  • The authors discussed briefly the problematic of having invalid queries which could fool the model. What about incoherence in R or T like having the wrong DNA for a given taxa or a mislabelling of branches in T? Would the algorithm be very much affected or can it still retrieve coherent results using the rest of the data? This could be particularly interesting if in the future AI is used to aggressively map the genome of species and could possibly make some mistakes here and there.
  • The algorithm was trained for 40 epochs without data augmentation, yet it does not show the training curves. In classical DNN this is likely to lead to severe overfitting - did the authors verified what is going on there and whether the results could be improved by adding regularizers in the loss / data augmentation?
2 comments

We thank the reviewer for the questions. Both led to new experiments.

Training process. Appendix B.2 now reports the hyperparameters and where they come from. They were carried over unchanged from an earlier campaign and are the same for all four markers. We ran no search and do not claim that they are optimal.

Q2, overfitting over 40 epochs. Figure B.2 now shows the training curves of the four markers. The held-out loss reaches its minimum between epochs 14 and 34 and then stays within 0.01 to 0.07 of it until epoch 40: we see no overfitting. We also tested the one augmentation that can act on the network's inputs: masking 10% or 20% of the retrieved sequences during training moves no metric by more than 0.5 points (Table B.1; ITS, one seed).

Q1, errors in the reference and the hierarchy. Appendix H now tests both on ITS, with one seed. With 5% of reference sequences relabelled to another genus, the network loses 0.5 points of genus accuracy and 1.1 of species accuracy on known-species queries, versus 4.6 and 3.5 for best-hit on the same retrieved sequences: reading 20 sequences absorbs a wrong label that the closest sequence alone passes on. With 5% of the hierarchy's branches moved to another parent, routing rate and coverage change by less than one point (Table H.1). This noise is uniform and synthetic; real errors cluster, and Appendix H and Section 6 keep that caveat.

Reviewer_Qw7uOpenReview ↗

Answer to rebuttal

I thank the authors for their additional experiments. I raised my confidence.

Reviewer_hL1XOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper notes that DNA barcode classifiers can only output species that have a reference sequence, which leaves out most named species. It proposes TaxoSet, which retrieves the nearest reference sequences and, when a small network flags the query as novel, returns the unsequenced relatives of the matched taxa from the full taxonomy as a candidate set. The paper is not clear to read and understand at all and is presented extremely poorly. Please see weakness 1 for more details.

Critical strengths
  1. The paper aims at a practical problem. For eDNA biodiversity surveys, a candidate set that contains the true genus a majority of the times should be more useful than a confident wrong class or not naming a class at all.
Critical weaknesses
  1. Clarity is a major obstacle to evaluating this paper. The abstract does not describe the method. It never states the pipeline (k-NN retrieval, candidate parents from the hierarchy, a set encoder scoring candidates, a threshold choosing between a single name and a set). Some sentences in the abstract are simply not interpretable, even after reading the paper, e.g. “At species rank it falls to 28%, where the limit is detecting novelty rather than the set itself”. Much of the paper is written as “X, not Y” (e.g., “informational, not computational,” “a calibrated refusal, not a silent error,” “headroom, not accuracy”). Individually these are fine. In aggregate, the reader is repeatedly told what a result is not and must infer what it actually is. Please state claims directly, with the number they rely on. The paper also uses inconsistent and overloaded terminology. The same quantity appears as “end-to-end coverage” and “joint coverage". The returned set is the “full set”, “complete set”, and “full elimination set”. The novelty decision is a “gate”, a “threshold”, and “routing”. Several terms also collide with established usage: “open-reference” already has a specific meaning in metabarcoding (open-reference OTU picking). Please fix one name per concept, define it once, and consider renaming the colliding terms. Most paragraphs of Section 4 state a conclusion and defer the evidence to Appendices C–G.

  2. Learning does not contribute much to the method. Retrieval and the hierarchy alone reach 0.721/0.851 coverage on the external benchmark (TaxoSet: 0.695/0.822), and the network beats unlearned rules in 18 of 27 cells, often narrowly. What does the learned component add beyond calibrated routing?

  3. PROTAX already scores unsequenced taxa by splitting probability among siblings. Could you clarify your novelty above PROTAX?

  4. The comparison with published methods is unbalanced by construction. Published classifiers score zero on absent taxa because they cannot output those names, while TaxoSet returns sets of dozens to hundreds of names. The resulting contrast (0 vs. 0.82 on ITS) reflects the output format more than the method.

Questions for authors

Please see weaknesses.

4 comments

We thank the reviewer for the comments. We have thoroughly revised the manuscript to improve its clarity and readability and conducted the additional comparative analyses requested by the reviewer.

W1, clarity. We apologize if certain parts of the original submission were not clear enough. In the revised manuscript, we have rewritten the Abstract, Introduction, and Sections 2–3 to improve clarity and ensure consistent terminology, as requested by the reviewer. The coverage table is now in the body (Table 3).

In short: each concept is now defined once, and each result is given with its number and its table.

W2, what learning adds. The reviewer raises an important point: retrieval and the hierarchy produce the candidate names without training. Following this comment, the revision measures what learning adds. Against three fixed rules given identical retrieved sequences, hierarchy and false-positive target, TaxoSet wins 15 of 27 paired comparisons and loses 3 on our leave-clade-out settings, and it wins at every budget for novel genera on all three markers (Table D.7). On ITS novel genera at ten names, it reaches 0.568 coverage versus 0.503 for the best fixed rule (Table D.6). Section 6 states two limits: an identity threshold calibrated the same way routes as well as the learned decision (Appendix D.7), and on the external benchmark the best fixed rule covers as much as TaxoSet or more (Table G.4).

In short: when only a few names can be returned, learning brings a measurable gain over the best fixed rule on identical inputs.

W3, novelty over PROTAX. We thank the reviewer for the opportunity to clarify the relation to PROTAX, the closest prior work, which Section 2 now credits explicitly for assigning probability to unsequenced taxa. TaxoSet differs on three points: it returns a prediction set evaluated on coverage and size; it chooses between a reference-backed name and that set with a threshold calibrated at a stated false-positive target; and it reads only retrieval scores, with no profile alignment. On the external ITS benchmark, under the same output format, PROTAX reaches 0.834 coverage with a median of 347 names; TaxoSet reaches 0.818 with 37, and at ten names it reaches 0.495 coverage versus 0.226 for PROTAX, whose expanded names are taken in random order (Table G.3). With identical retrieved sequences, hierarchy, splits and false-positive target, TaxoSet has higher coverage than our PROTAX adaptation in all nine settings, eight after Bonferroni correction (Tables C.3 and C.4). We also retrained the released PROTAX-GPU engine: on 12S novel genera it routes 0.047 of queries with the correct parent, versus 0.520 for TaxoSet; on COI the differences are within the engine's seed variation (Table C.2).

In short: PROTAX introduced probabilities for unsequenced taxa. TaxoSet turns them into calibrated prediction sets that, on ITS, are about nine times smaller at similar coverage, and it covers more than our PROTAX adaptation in all nine controlled settings.

W4, comparison format. Following the reviewer's suggestion, the comparison is now made under one output format: Table G.3 expands each published method's predicted parent to its unsequenced genera, at the same false-positive target. On ITS at ten names, TaxoSet and IDTAXA have the highest coverage of the thirteen methods (0.495 and 0.492); the three methods with higher full-set coverage return median sets of 347 to 4,324 names, against 37 for TaxoSet. On COI, EPA-ng does better: 0.737 coverage with 59 names versus 0.698 with 119, and 0.257 versus 0.235 at ten names. Section 6 states this limit.

In short: with every method given the same output format, TaxoSet remains at the top on ITS at ten names and is behind only EPA-ng on COI.

Thank you for the revision. However, the new analyses in the revision reinforce my central concerns rather than resolve them.

1. The new results show that the learned component adds little or nothing over fixed rules for the prediction sets, the paper's central output.

  • On the external benchmark, where novelty is real rather than simulated, the new Table G.4 shows that best-hit expansion with an identity threshold matches or exceeds TaxoSet at every name budget: 0.508 vs. 0.495 at ten names on ITS, and 0.503 vs. 0.424 at fifty names on COI. The identity threshold also routes absent genera at the same rate as the learned decision (0.970 vs. 0.967 on ITS, 0.970 vs. 0.972 on COI). The paper itself now concludes that the learned decision and ranking "add nothing measurable" there.
  • At the full prediction set, Appendix D.7 states that whether the learned novelty score beats a calibrated identity threshold "stays open."
  • On the authors' own splits, under the revised evaluation, the count of wins over fixed rules is 15 of 27 (down from 18), with 3 losses.

The measurable benefit of learning for prediction sets is thus confined to the authors' own leave-clade-out splits under a name budget. For an ICLR submission whose method is the learned network, this is a central issue.

2. The revision confirms that the core mechanism comes from PROTAX. The revised Section 2 now states that Equation 1, the uniform allocation among unsequenced siblings, is adopted "as PROTAX does." The remaining differences listed in the revised Section 5 are:

  • returning the tied group as a set, which is an evaluation convention;
  • a calibrated threshold, which is not shown to outperform an identity threshold;
  • a learned scoring function, which a fixed rule matches on the external benchmark.

The rebuttal adds that TaxoSet needs no profile alignment. But as Section 5 notes, PROTAX itself takes user-supplied similarity such as BLAST, so this distinguishes TaxoSet from the PROTAX-GPU release rather than from the method. The abstract's PROTAX comparison is against the authors' own restricted adaptation. The released PROTAX-GPU engine, retrained on the same data, is within seed variation of TaxoSet on COI, where both are below the best-hit expansion fixed rule (Table C.2).

3. The matched-format comparison shows that the headline contrast was largely an artifact of output format, as noted in my original review. The original submission's main external result was that published methods score zero on absent genera. Once those methods are given the same output format (Table G.3), they are competitive. On COI, EPA-ng (freetree) covers more with smaller sets (0.737 with 59 names vs. 0.698 with 119), and EPA-ng (phyltree) leads at ten names (0.257 vs. 0.235). On ITS, IDTAXA is level with TaxoSet at ten names (0.492 vs. 0.495). The revision also corrects two claims of the original submission: TaxoSet does not rank first on COI (BLAST top-hit scores higher), and no published method in fact recovered the four COI queries originally reported.

4. The rebuttal's summaries overstate the results. It says TaxoSet "remains at the top on ITS at ten names," but the authors' own fixed rule scores higher there (0.508, Table G.4). Its summary that learning brings "a measurable gain" when few names can be returned holds only on the authors' own splits. on the external benchmark, as the rebuttal's own text acknowledges, the fixed rule matches or exceeds TaxoSet at every budget.

In summary, the new experiments show that the contribution rests on an existing allocation principle plus candidate expansion and thresholding. Fixed rules match the learned components on the external benchmark and at the full prediction set, and the remaining gains appear only on the authors' own splits. I therefore maintain my recommendation to reject.

We thank the reviewer for the detailed reading of the revision. The figures the reviewer cites are accurate, and they narrow what we can claim for learning. We start with these points.

On the external benchmark. The reviewer is right that a fixed rule scores higher than TaxoSet on ITS at ten names (0.508 versus 0.495, Table G.4). Our summary should have said "among published methods", as Section 6 of the paper does. We built this rule within the formulation of the paper. In the benchmark, the predictions published for every method name no genus absent from the reference. Among the published methods given the same output format, TaxoSet is first on ITS at ten names, with IDTAXA at the same level (0.495 and 0.492), and second on COI behind EPA-ng (Table G.3). The strength of the fixed rule is a limit of the learned network.

On EPA-ng. EPA-ng is the only published method ahead of TaxoSet, on COI (0.257 and 0.248 for two of its variants, versus 0.235). It needs a multiple alignment, which the benchmark provides for COI and not for ITS. Such an alignment is hard to build for a marker like ITS, whose length varies across lineages. We ran it on ITS ourselves as an exploratory check. With the better of the two query alignments we tried (MAFFT and a profile HMM), it reaches 0.595 coverage and 0.333 at ten names, against 0.818 and 0.495 for TaxoSet. TaxoSet needs no multiple alignment and runs unchanged on both markers.

On PROTAX. PROTAX assigns probability to taxa without reference sequences, and Section 2 credits it for the uniform allocation among unsequenced siblings. The released PROTAX-GPU engine gives a probability to each placement and is evaluated on its most probable one. TaxoSet returns the unsequenced names as a set, and the paper evaluates this set by its coverage and its size. Given the same output format on ITS, TaxoSet returns a median of 37 names for 0.818 coverage, against 347 names for 0.834 for PROTAX. At ten names, TaxoSet covers 0.495 on ITS and 0.235 on COI, against 0.226 and 0.153 for PROTAX (Table G.3). TaxoSet is also reused across markers without retraining, whereas a PROTAX model is trained for one marker and its taxonomy. The reviewer is right that the absence of a profile alignment separates TaxoSet from the PROTAX-GPU engine only.

On our own settings. The reviewer is right that our summary of a "measurable gain" should have been limited to these settings, as Section 6 of the paper does. These settings use large references, with 970,750 ITS sequences against 10,776 in the benchmark. Against the best fixed rule, TaxoSet wins 15 of 27 paired comparisons and loses 3, and 9 are undecided. A win or a loss is an interval that excludes zero, and 11 wins and the 3 losses survive Bonferroni correction. TaxoSet wins at every budget for novel genera on all three markers (Table D.7). On COI novel genera at ten names, the best fixed rule reaches 0.030 and TaxoSet 0.071.

On names present in the reference. Learning also acts on these names. The network is ahead of best-hit on identical retrieved sequences in seven of eleven settings (Table 2). Reviewer Qw7u asked us to test the effect of labelling errors. With 5% of the ITS reference relabelled to another genus, best-hit loses 4.6 points of genus accuracy on known-species queries, and the network 0.5 (Appendix H, one seed). This test says nothing about novel names.

What we claim. The contribution is the open-reference formulation. Candidate names come from retrieval and the hierarchy, and they include names absent from the reference. A decision calibrated at a stated false-positive rate returns them, and the result is evaluated by coverage together with set size. Prior work assigns probabilities to such names (PROTAX) or reports a new branch (BayesANT). To our knowledge, none returns these names as a prediction set evaluated in this way. The network is trained on references from which whole clades were removed before retrieval, so the rank at which a query becomes novel is a label it learns. The learned network is one way to implement the formulation, and we do not claim that it beats every rule.

What remains open. The network is simple and reads little, only similarity scores. Its role in the paper is to show that, within this formulation, a learned model can exploit this kind of information. We expect that adding other information to its input could let the approach work better, and we have not tested this. Where a fixed rule is enough, the paper says so (Section 6). The formulation is what makes simple rules and learned models comparable on a task that published methods have not yet addressed in this form.

I thank the authors for their response. It largely confirms the concerns I raised earlier. On the external benchmark (Table G.4), a fixed rule performs as well as or better than TaxoSet. The benefit of learning is only demonstrated on the authors' leave-clade-out splits. Also, the lack of profile alignment is a difference from PROTAX-GPU, not PROTAX itself. The authors now clarify that their main contribution is the open-reference formulation, with the learned network being just one possible implementation rather than an improvement over fixed rules.

My main concern remains the same. The central idea of distributing probability uniformly among unsequenced siblings already comes from PROTAX. The other components, such as returning tied groups as sets, using a calibrated threshold, and reporting coverage alongside set size, can also be implemented with fixed rules. Based on the external benchmark, the proposed network does not offer a clear advantage over these simpler alternatives. The EPA-ng result on ITS is interesting, but it is exploratory and was not part of the submitted paper. Similarly, the points about PROTAX's set sizes and transfer across markers do not explain why a learned model is needed, since they apply to fixed rules as well.

I appreciate the authors' willingness to clarify their claims. However, I still do not see enough methodological novelty to justify acceptance at the ICLR main track. I therefore maintain my recommendation to reject.

Reviewer_aZKNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper tackles open-reference taxonomic assignment for DNA barcodes: instead of forcing every query into a reference-backed label, it retrieves top-k hits, uses the taxonomic hierarchy to expose candidate names that are absent from the sequence reference, and returns a calibrated set-valued prediction​ plus a novelty decision. The core idea is clean: (1) model the query as an "episode" of retrieved evidence; (2) score candidates through their parents and distribute probability uniformly over unsequenced siblings (Eq.1); (3) a small DeepSets network (0.34M params) learns routing between reference-backed names and hierarchy-backed sets. Evaluation uses strict leave-clade-out splits across ITS/12S/16S/COI, plus a temporal GTDB rollback and an external benchmark. Results show meaningful gains over best-hit, vote, PROTAX adaptations, and classical baselines on novel-genus/novel-family coverage, while staying honest about set-size blow-ups (e.g., COI median 1,493 names).

Critical strengths
  1. Honest about the info-theoretic ceiling: even with correct parent, single-name accuracy ≤ 1/|U_g|.
  2. Leave-clade-out + GTDB r207 rollback is more realistic than standard zero-shot splits.
  3. Small network (0.34M) beats 11× larger attention in 8/9 settings — good evidence the gain is from problem framing, not capacity.
  4. External TaxClass results are striking: prior methods recover ~0 absent genera; TaxoSet gets 0.695–0.822 coverage.
  5. Marker-transfer experiment (no weight change) is a strong generality check.
Critical weaknesses
  1. Table 2 / Fig 2 report coverage but median |U| ranges 11 (ITS) to 1,493 (COI). A coverage of 0.71 with 1.5k names is not actionable in curation. The paper needs a primary metric like coverage@b with b fixed (already in Appendix D.6, but secondary). 2.Retrained PROTAX-GPU is fair, but ITS is excluded from the released-engine comparison due to alignment failure (34 columns). The top-20 adaptation is reasonable, yet a reader may wonder if PROTAX’s poor ITS showing is mostly alignment, not modeling. Authors mitigate this well in C.2, but the caveat belongs in the main baseline paragraph.
Questions for authors

1.The species-rank gate collapses on 16S (Table A.2). Would you recommend deploying TaxoSet without a gate at species rank, or abstaining? 2.Retrieval is VSEARCH-only in the main pipeline. Given D.5 shows search-depth helps ITS, is the 20-hit cap a deployment choice or a network constraint? Did 50-hit retraining actually fail to help, or just marginally?

1 comment

Thank you for all the suggestions. Each point led to a change in the paper.

W1, coverage at a fixed budget. The reviewer raises an important point: full-set coverage alone hides the size of the set. The coverage table is now in the body (Table 3) and reports coverage at ten names beside full-set coverage and set size. On ITS novel genera, TaxoSet reaches 0.694 with a median of 11 names and 0.568 at ten names. On COI novel genera it reaches 0.709 with a median of 1,493 names and 0.071 at ten, and the body now gives both figures together. Table D.1 gives budgets from 1 to 50 names.

W2, PROTAX on ITS. This is a fair point, and Section 4.1 now states this caveat. On ITS the profile alignment of the released engine keeps only 34 columns (Appendix C.2), and the ITS comparison rests on our adaptation alone. Where both run, the adaptation is the weaker of the two: on COI the retrained engine is within seed variation of TaxoSet while the adaptation is far below. Appendix C.2 says that the ITS result carries this caveat.

Q1, species rank on 16S. 16S has one sequence per species, which leaves its species rank without a threshold. The best option there is to abstain at species rank and return the genus-level prediction set, and Section 6 now says so. The genus-rank threshold can be calibrated, at a cost: coverage of novel genera is 0.25 to 0.29 with it, versus 0.87 to 0.92 before it (Tables A.1 and A.2).

Q2, 20 retrieved sequences. Following the reviewer's question, we tested longer lists. At 50 sequences the retrieval bound rises from 0.813 to 0.863 on ITS novel genera, but retraining all four markers at 50 did not raise reference-backed accuracy (Table D.4). The network accepts any list length, so 20 is a choice of cost that these runs support. We did not measure prediction-set coverage at 50.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response and corrections

We thank the three reviewers. The revised PDF is uploaded; changes are coloured by the review they answer (legend on the title page). Each reply points to the section or table that answers the point.

Corrections we found ourselves. While preparing the revision, we also corrected three points.

  1. External benchmark. We rebuilt it from its repository. Three published methods were missing from our September files. With them, BLAST top-hit (0.796) is ahead of TaxoSet (0.792) on COI genera present in the reference: TaxoSet is second of sixteen, where we had said first. The "four of 3,780" queries on which a published method named an absent genus are zero. The other benchmark cells moved by a few thousandths.

  2. Comparison with fixed rules. We regenerated the analyses with the released code; the September files predated the last fixes to the evaluation code. The count is now 15 wins, 3 losses and 9 undecided cells (it was 18, 3 and 6), with 11 wins surviving Bonferroni (it was 12). Coverage moved by at most 0.01. The realized false-positive rate ranges from 0.045 to 0.070 for a 5% target (it was 0.041 to 0.051); the highest value is COI novel families (Appendix D.3). The paired table no longer has an attention-encoder column; Table B.2 keeps that comparison.

  3. Recall@20 (Table D.2). Recomputed with the script behind the other recall tables; cells moved by at most 0.010 (93.1% for known ITS species, it was 93.3%).

The conclusions are unchanged, except on the external benchmark, where the revision now states that a fixed rule does as well as the learned method (reply to hL1X, W2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.