- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes DeltaGenome, a framework designed to accelerate GFM inference by reusing computation between original sequences and slightly modified variants. The authors analyze how modifications affect the hidden representations and what parts can be reused in terms of inference. Results show that on four GFMs, those modifications are structured and predictable. But only HyenaDNA shows a 1.33x speedup, while others get little improvement or are even slower.
Critical strengths
- The motivation of this paper is interesting and practically relevant. Reusing representations between slightly changed sequences is a promising direction for accelerating inference.
Critical weaknesses
- Most figures in this paper are small in the text and have lots of dense visual elements, this makes them very difficult for readers to read and understand. Especially for Figure. 1, I can hardly read the text without zooming in a lot. Also, considering that key information in Figures. 2-5 is relatively limited, it would be better to use tables rather than those figures with tiny text and excessive whitespace to report the results. I would suggest the authors improve the readability of this paper.
- This paper could be better organized. The key idea of this paper is to accelerate inference for genomic foundation models, so those results are expected to be presented earlier. But in the experimental section, 5.1-5.3 are talking about perturbation, prediction and reconstruction, and main acceleration results are only stated in Section 5.4 on page 8. I think the experimental narrative does not align well with the paper's primary objective.
- For all four GFMs tested in the paper, only HyenaDNA shows a 1.33x speedup using the proposed method. The other three models NT-v2, DNABERT-2 and Caduceus get little improvement or are even slower. Considering that inference acceleration is the main motivation for this paper, the current results do not show practical significance or general applicability.
Questions for authors
- Can the authors provide the results on the most recent large-scale GFMs like GenomeOcean[1] and Evo 2[2]? All four models tested in the paper are relatively old and may not represent SOTA GFMs. Those generative models can also do classification, and since they are large in parameters, the inference acceleration on them is potentially more valuable. I would suggest the authors add results on those models, especially Evo2.
[1] Zhou, et al. "Genomeocean: An efficient genome foundation model trained on large-scale metagenomic assemblies."
[2] Brixi, et al. "Genome modelling and design across all domains of life with Evo 2."
Thank you for the practical motivation you identified and the specific suggestions on readability, organization and contemporary large-scale GFMs. I have substantially revised both the presentation and the experiments.
Presentation and organization. Figure 1 is redesigned as a simpler conceptual overview. The main text now has three figures and three tables: Table 1 summarizes mechanisms, Table 2 reports absolute latency, speedup and numerical fidelity, and Table 3 summarizes perturbation geometry. Dense diagnostic plots and detailed tables are moved to the appendix. The Results now begins with matched acceleration (§5.1), followed by amortization/state organization (§5.2) and contemporary-GFM execution (§5.3); perturbation prediction and online utility follow in §5.4, and downstream prediction preservation concludes §5.5. The abstract and introduction prioritize reference-aware execution across eight backbones, with the harmonized 384-SNV analysis clearly restricted to the original four. This addresses the original mismatch between motivation and experimental narrative.
Practical significance. The revised comparison separates full-system gains from incremental reuse. At K=128, HyenaDNA has 2.042× native-relative and 1.107× compiler-matched prepared-system speedups, NT-v2 1.003× and 1.017×, Caduceus 1.762× and 0.979×, and DNABERT-2 0.674× and 0.663×. The gains for HyenaDNA and NT-v2 are consistent across all eight references; their descriptive 95% intervals are [1.1033,1.1113] and [1.0164,1.0167]. Absolute timings, repeat variability, checkpoint identities, numerical errors and matched regional operators are reported in Table 2A and Appendices D–E. Caduceus's shared directional frontier reduces K=128 cache tensor storage by 79.2× relative to coordinate-specific caching, not peak GPU memory; it does not yield a matched latency gain. An independent HyenaDNA prefix comparison is also reported and remains competitive. The revision therefore does not attribute all native-relative acceleration to reuse.
GenomeOcean, Evo2 and other recent models. As requested, the study includes GenomeOcean-4B and Evo2-7B, as well as NT-v3 and Carbon-3B. At K=128, Evo2-7B achieves 1.292× native-relative and 1.016× matched setup-inclusive speedups, with wall totals of 98.022, 77.053 and 75.867 s for native, optimized dense and reuse. All 512 requests are bitwise exact. NT-v3 achieves 1.316× native-relative and 1.105× matched setup-inclusive CUDA speedups; a separate audit verifies bitwise equality on 512 requests. GenomeOcean-4B achieves 1.152× on 512 prepared eligible requests, with bitwise-exact outputs; a separate mixed workload with only 109/512 eligible requests yields 0.797× aggregate wall-clock speedup while preserving bitwise equality. Carbon-3B shows 1.022–1.053× warm gains across three edit-position strata, without setup-inclusive measurements. These results establish applicability of concrete reuse mechanisms beyond HyenaDNA, but not universal acceleration or controlled model-size scaling (Table 2B, Fig. 3, §5.3, Appendix F/Tables 15–19). Four of the six backbones evaluated against matched optimized dense retain positive setup-inclusive gains at K=128.
Downstream classification. To address whether reuse preserves biological predictions, the revision adds a HyenaDNA pathogenic/common variant classifier trained on frozen representations from 38,619 pairs, with an identical head across dense, independent prefix and DeltaGenome execution. Both reuse paths preserve all 1,018 held-out class decisions; DeltaGenome matches dense AUROC 0.644886, average precision 0.582140 and accuracy 0.592338. The natural test workload has 987 singleton groups among 1,002 references, yielding aggregate speedups of 0.725× (DeltaGenome) and 1.077× (prefix). In controlled unlabeled panels with greater sharing, DeltaGenome achieves 1.169× median paired speedup at K=128. These results validate prediction preservation and the importance of reference sharing, not improved classification accuracy. Classification on GenomeOcean/Evo2 is not claimed (§5.5, Appendix K, Fig. S3, Tables 26–28).
I appreciate the suggestions that led to a more readable execution-first paper and a substantially broader, more rigorous evaluation of modern genomic models.