- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents TAPFM, a framework for jointly adapting a large vision transformer and an attention-based multiple instance learning (MIL) model on a single GPU. The paper analyzes GPU memory consumption, including model parameters, gradients, optimizer states, and activation memory, and evaluates three adaptation strategies using GigaPath. The first strategy applies LoRA to all transformer blocks. The second freezes the early transformer blocks and fully fine-tunes only the later blocks. The third combines these approaches by freezing the early blocks and applying LoRA only to the later blocks. Experiments are conducted on a single NVIDIA H200 GPU using two mutation-prediction tasks: BLCA-FGFR3 and LUAD-EGFR.
The paper provides a useful engineering analysis of memory usage and demonstrates the feasibility of adapting a large pathology foundation model with a slide-level MIL model on a single high-memory GPU. However, the proposed strategies are primarily combinations of existing techniques, and their technical novelty is limited. The experimental evaluation is also restricted to one foundation model, two prediction tasks, one high-memory GPU, and a single random seed. In addition, differences in tile budgets across configurations make it difficult to determine whether the observed performance differences result from the adaptation strategies or the amount of input information.
Critical strengths
- The paper addresses a computational challenge in digital pathology, where whole-slide images contain a large number of image tiles, and jointly fine-tuning a large pathology foundation model and a slide-level MIL model can require substantial GPU memory.
- The distinction between activation memory and memory consumed by model parameters, gradients, and optimizer states is useful.
- The proposed hybrid strategy increases the reported tile budget from 100 to 1,000 tiles per WSI on a single H200 GPU while substantially reducing the number of trainable encoder parameters.
- The paper includes external evaluation using TCGA and an additional institutional cohort. This is a useful aspect of the experimental design and provides some evidence of performance beyond a single dataset.
- The paper is relatively transparent about the trade-offs between computational efficiency and predictive accuracy. The comparison among different adaptation strategies provides practical insights into selecting fine-tuning configurations under memory constraints.
- The paper provides useful implementation details, including LoRA rank, target attention projections, and tile chunking strategies. These details improve reproducibility and may help researchers implement similar memory-efficient training pipelines.
Critical weaknesses
- The technical novelty of the proposed method is limited. LoRA, freezing early transformer layers, and centered kernel alignment (CKA) are all established techniques. The proposed framework primarily combines these existing methods rather than introducing a fundamentally new algorithm or optimization strategy.
- The three proposed adaptation strategies appear to be engineering configurations rather than three distinct methodological contributions. Although their comparison is useful from a practical perspective, the paper does not clearly establish what new technical insights or algorithmic contributions distinguish TAPFM from existing parameter-efficient fine-tuning approaches.
- All experiments are conducted using a single pathology foundation model, GigaPath, while the paper makes broader claims about pathology foundation models and pretrained vision transformers.
- It is unclear why the proposed adaptation strategies are particularly suitable for pathology foundation models. From the technical description, the methods appear largely applicable to general vision transformers. The paper does not sufficiently explain which challenges are unique to pathology foundation models or how the proposed strategies specifically address these challenges.
- The experiments are performed on a single NVIDIA H200 GPU. This is a high-end and expensive GPU with substantially more memory than commonly available GPUs. Therefore, the results demonstrate that the framework works on one high-memory GPU, but they do not establish whether it makes foundation model adaptation accessible on more commonly available single-GPU hardware.
- The experimental comparisons are potentially confounded by differences in the number of tiles used across configurations. When different methods are evaluated using different tile budgets, it becomes difficult to determine whether performance differences arise from the adaptation strategy or from the additional image information available to the model.
- Ideally, the authors should report results across multiple random seeds, including mean performance, standard deviations, and confidence intervals where appropriate.
- Although the memory decomposition is conceptually useful, some of the notation and assumptions are not sufficiently clear.
- The ability to process more tiles per WSI does not necessarily imply a reduction in overall training cost. Increasing the tile budget may increase computation, training time, and GPU utilization even when peak memory consumption is reduced.
- The paper states that the hybrid configuration enables "full-dataset training with 1,000 tiles per WSI." However, the training procedure still samples 1,000 tiles from each WSI rather than necessarily using all available tissue tiles. Therefore, this statement may be misleading if readers interpret it as processing every tissue tile in each whole-slide image.
- The ability to process 1,000 tiles per WSI does not automatically demonstrate that the additional tiles provide more clinically relevant information. Although a larger tile budget may improve tissue coverage, the paper does not clearly establish whether this translates into consistent improvements in predictive performance or clinical utility.
Questions for authors
- What is the main technical novelty of TAPFM beyond combining existing techniques such as LoRA and layer freezing?
- What challenges specific to pathology foundation models does the proposed method address?
- Can the authors provide comparisons using the same tile budget to ensure a fair evaluation of different adaptation strategies?
- Can the method work on more commonly available GPUs with 24 GB or 48 GB of memory, rather than an H200?
- Does processing more tiles consistently improve prediction accuracy, and how does it affect overall training time and computational cost?