- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors propose to estimate per-taxon relative abundance from DNA metabarcoding of bulk arthropod samples with a neural network built around the steps of the lab process. It then reverses this relationship by dividing the observed reads by a learned read yield per specimen. Two frozen external estimators supply the per-BIN inputs: (1) a body-size estimator built from segmenting 640k BOLD specimen images and fine-tuning BarcodeBERT to predict size from the barcode, and (2) a primer binding-efficiency estimator that scores primer sites retrieved from full-length barcodes and mitogenomes, with ridge-regression imputation for the rest. Separate heads predict body mass and presence from BarcodeMamba embeddings, taxonomy and bottle metadata. A self-attention module over all BINs in a bottle predicts the amplification term, and a small learned network merges the three PCR replicates. The model is trained with a negative-binomial likelihood plus a Jensen–Shannon divergence (JSD) composition loss, a presence BCE loss and a reads-per-specimen loss. The authors also contribute a paired dataset processed both by metabarcoding and by individual barcoding. The framework is evaluated on in-domain random splits, on spatial, temporal and sequencing-batch shifts, and on a small external dataset from another lab.
Critical strengths
- S1: The dataset is very valuable, which is more than 10x larger than previous existing datasets.
- S2: Model is derived from the lab process as a clear example of domain knowledge shaping the inductive bias.
- S3: Each component is motivated by data analysis.
- S4: Careful, fair tuning of baselines.
- S5: Broad evaluation under distribution shift.
- S6: The 30-block ablation, paired seed-for-seed with a noise band, shows that data and chemistry matter most.
Critical weaknesses
- W1: Small gains over generic deep baselines.
- W2: Potential circular ground truth for the bottles with excess specimens. In bottles where abundant morphospecies were counted by eye, those counts are assigned to BINs using the predictions of a model trained on the other bottles.
- W3: Weak inputs for primer efficiency. The reverse primer efficiency is imputed for 98.8% of BINs, from a predictor with = 0.16 whose outputs are heavily shrunk toward the mean. The forward imputer is worse than a taxonomic mean.
- W4: Statistical testing ignores variation between test bottles.
Questions for authors
- Q1: Can you report paired bootstrap confidence intervals over test bottles (not only over seeds) for the JSD and MAE differences between BIOTRAM and the best baseline in Table 1?
- Q2: How were the development weights (8, 0.15, 0.05) chosen, and on which model? If they were tuned on BIOTRAM, how should Table 3 be read as an "architecture-only" comparison?
- Q3: Please reconcile the presence AUC values (0.9136 in Table 1 vs. 0.9213 in Appendix O).
- Q4: How does performance change if BINs found by individual barcoding but missed by metabarcoding are kept in the evaluation (predicted as zero)? How many bottles hit the 768-BIN truncation, and what share of individuals does truncation remove?
2 comments
Reviewer 9yub individual reply: test bottles variation, development loss weights, presence AUC, missed BINs and BIN truncation
W4 & Q1 : Testing ignores variation between test bottles.
We agree with your point: with only 89 test bottles, the submitted results could depend on which bottles were randomly allocated to the test set. The revision replaces this split with nested 5-fold cross-validation over all bottles with complete counts (point 1 of our general response); this goes beyond what you asked for and gives a better estimate of performance.
We have also computed the paired bootstrap CIs over test bottles: Over the 419 test bottles (10,000 resamples), BIOTRAM minus each baseline with the recorded total is, for JSD: −0.0016 (95% CI −0.0033 to +0.0000) against the Transformer, −0.0013 (−0.0029 to +0.0002) against the 1-D CNN, −0.0041 (−0.0063 to −0.0020) against the MLP and −0.0051 (−0.0080 to −0.0027) against DeepSets. For MAE per bottle the interval excludes zero against every baseline (Transformer −0.027, −0.044 to −0.013), and so it does for AUC (Transformer +0.0057, +0.0029 to +0.0087). The JSD lead over the Transformer and the CNN is therefore small and not confirmed by the bootstrap, while the MAE per bottle and AUC leads are. The appendix gives these intervals, together with a two-level bootstrap that also resamples the runs of each bottle.
Q2: How were the development weights (8, 0.15, 0.05) chosen?
In the revision, every model’s hyper-parameters, loss weights included, are tuned inside each cross validation fold on validation bottles only, so no reported comparison uses weights chosen during BIOTRAM’s development. In the submission, these weights were chosen on BIOTRAM's validation set, during its development, and not updated afterward. The submitted Table 3 used this one setting for every model, so that only the architecture differed, while the main comparison (the submitted Table 1) tuned each model’s three weights separately, with the same protocol and budget (App. O); none of the five models, BIOTRAM included, selected (8, 0.15, 0.05). The architecture-only comparison (Table 3) now gives every baseline BIOTRAM’s tuned weights of the same fold, so that only the architecture differs: BIOTRAM’s JSD is then 0.0017 below that of the strongest baseline, against 0.0014 when each model has its own weights (App. I).
Q3: Presence AUC, 0.9136 against 0.9213.
Both values are right but measure different things; point 8 of our general response explains them and how the revision reports them. We thank the reviewer for making us aware of this inconsistency.
Q4a: Missed BINs.
We added this analysis (new App. U); point 7 of our general response gives the results.
Q4b: 768-BIN truncation.
This was an error in the text from a preliminary version of the model. The version of the methodology used in the submitted version did not have a 768-BIN limit. Every model was trained and scored on every BIN of every bottle. We apologise for this mistake and have now corrected the text accordingly.
Edit: Re-organized content.
Reviewer 9yub individual reply: gains over baselines, incomplete ground truths and primer-efficiency input
Thank you for reading and giving us feedback on the paper. We respond to each of your questions and weaknesses that you identified below. Additionally, please see our general response which describes changes in the evaluation protocol, affecting all tables, and answers in full the concerns that several reviewers raised.
W1: Small gains over generic deep baselines.
We agree that the gap between the BIOTRAM architecture and the other deep baselines is only modest; point 4 of our general response explains what this comparison measures and how it relates to the larger gain over existing methods.
W2: Circular ground truth for the excess bottles.
The text about excess that raised the concern about circularity is wrong and we apologize for it: no label is completed by a model. Point 6 of our general response gives the details.
W3: Weak primer-efficiency inputs.
Your point is right and we, in fact, struggled to estimate the primer binding efficiency for the metabarcodes. The consensus sequences that we get after amplification are unfortunately a bad predictor for the binding sequences, outside of its range and we didn't have enough sequences that cover the primer binding sites (mitogenomes and full-length barcodes) to get a good coverage of the taxonomy, in particular for the reverse site efficiency. This is a current limitation of the model and we hope to obtain longer sequences in the future to mitigate it. Regarding your comment on the taxonomic mean, while it does indeed perform better than our imputer, it relies on the BIN’s full taxonomy at genus level or below, whereas our imputer needs only the amplicon. This matters because 77% of BINs in our metabarcoding dataset have no known genus name, so we cannot use it on the majority of the data. The data we fit the reverse primer efficiency model on has full taxonomic data, so we were able to measure the performance of the taxonomic mean as a baseline. To better show the effect of incomplete taxonomic labels, we have expanded this analysis to also show the performance of the mean for truncated taxonomy up to a specific rank (species, genus, family, order). With the taxonomy known to genus or species, the taxonomic mean predicts the forward efficiency slightly better than our barcode predictor (R2 0.261 against 0.234); known only to family it is slightly worse (0.229), and to order it fails (0.043). For the reverse efficiency our predictor is better at every rank (0.162 against 0.095 at most). With each test BIN’s taxonomy cut to the ranks actually known for our BINs, the taxonomic mean reaches R2 0.229 (forward) and 0.087 (reverse), against 0.234 and 0.162 for our predictor. We also retrained BIOTRAM with the taxonomic mean in place of every predicted efficiency: its JSD does not change (−0.0001, 95% CI −0.0010 to +0.0008). We have also added discussion of this to our Limitations section, where we state explicitly that, while primer binding efficiency belongs in the model’s structure, current reference data cannot estimate it well; single-specimen barcodes that keep the primer sites, and more mitogenomes, would improve it.
Edit: Re-organized content.