acceptodds
Withdrawn from ICLR 2027

Freeing Activation Memory for Task Adaptation of Pathology Foundation Models: Layer-Selective Freezing and Partial Low-Rank Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Task Adaptation of Pathology Foundation Models (TAPFM, kumar2025tapfm) showed that a vision-transformer (ViT) pathology foundation model (PFM) and an attention-based multiple instance learning (MIL) aggregator can be jointly fine-tuned on a single GPU by detaching gradients between two separate computational graphs. This makes end-to-end adaptation practical without the multi-GPU clusters used by prior work, but the number of tiles that can be sampled per whole-slide image (WSI) per training step remains small (e.g., 100 tiles for GigaPath), because activation memory for backpropagation through a giant ViT scales with both the number of tiles and the number of transformer blocks. Motivated by the observation in TAPFM that task-adaptation gradients concentrate in the final transformer blocks with negligible updates to earlier layers, we translate this empirical finding into an actionable memory-saving mechanism. We introduce and compare three single-GPU strategies for GigaPath: (i) low-rank adaptation (LoRA) applied to all transformer blocks, which reduces gradient and optimizer-state memory; (ii) freezing all but the last transformer blocks, which additionally removes the frozen prefix from the autograd graph and eliminates its activation memory; and (iii) a hybrid that freezes all but the last blocks and applies LoRA only within those unfrozen blocks, combining both memory savings. Empirically, these strategies raise the maximum tiles-per-WSI-per-epoch achievable on a single H200 from TAPFM's original 100 to 300 (LoRA-only), 500 (freeze-last-N only), and 1000 (freeze-last-N + LoRA) with , respectively. We provide a memory-decomposition analysis that explains why LoRA alone does not reduce activation memory while layer freezing does, formalize the hybrid strategy as a generalization of TAPFM's detached dual-gradient update restricted to a block subset, and show across institutional and external cohorts that the proposed configurations retain strong FGFR3 and EGFR mutation-prediction performance while offering task-dependent trade-offs between predictive accuracy and resource efficiency. Our results suggest that removing activation-memory bottlenecks is a viable path toward more scalable and parameter-efficient single-GPU adaptation of large PFMs.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Pathology Foundation Models, Whole Slide Images, Mutation Detection, Vision Transformer, Multiple Instance Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GdMXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents TAPFM, a framework for jointly adapting a large vision transformer and an attention-based multiple instance learning (MIL) model on a single GPU. The paper analyzes GPU memory consumption, including model parameters, gradients, optimizer states, and activation memory, and evaluates three adaptation strategies using GigaPath. The first strategy applies LoRA to all transformer blocks. The second freezes the early transformer blocks and fully fine-tunes only the later blocks. The third combines these approaches by freezing the early blocks and applying LoRA only to the later blocks. Experiments are conducted on a single NVIDIA H200 GPU using two mutation-prediction tasks: BLCA-FGFR3 and LUAD-EGFR.

The paper provides a useful engineering analysis of memory usage and demonstrates the feasibility of adapting a large pathology foundation model with a slide-level MIL model on a single high-memory GPU. However, the proposed strategies are primarily combinations of existing techniques, and their technical novelty is limited. The experimental evaluation is also restricted to one foundation model, two prediction tasks, one high-memory GPU, and a single random seed. In addition, differences in tile budgets across configurations make it difficult to determine whether the observed performance differences result from the adaptation strategies or the amount of input information.

Critical strengths
  • The paper addresses a computational challenge in digital pathology, where whole-slide images contain a large number of image tiles, and jointly fine-tuning a large pathology foundation model and a slide-level MIL model can require substantial GPU memory.
  • The distinction between activation memory and memory consumed by model parameters, gradients, and optimizer states is useful.
  • The proposed hybrid strategy increases the reported tile budget from 100 to 1,000 tiles per WSI on a single H200 GPU while substantially reducing the number of trainable encoder parameters.
  • The paper includes external evaluation using TCGA and an additional institutional cohort. This is a useful aspect of the experimental design and provides some evidence of performance beyond a single dataset.
  • The paper is relatively transparent about the trade-offs between computational efficiency and predictive accuracy. The comparison among different adaptation strategies provides practical insights into selecting fine-tuning configurations under memory constraints.
  • The paper provides useful implementation details, including LoRA rank, target attention projections, and tile chunking strategies. These details improve reproducibility and may help researchers implement similar memory-efficient training pipelines.
Critical weaknesses
  • The technical novelty of the proposed method is limited. LoRA, freezing early transformer layers, and centered kernel alignment (CKA) are all established techniques. The proposed framework primarily combines these existing methods rather than introducing a fundamentally new algorithm or optimization strategy.
  • The three proposed adaptation strategies appear to be engineering configurations rather than three distinct methodological contributions. Although their comparison is useful from a practical perspective, the paper does not clearly establish what new technical insights or algorithmic contributions distinguish TAPFM from existing parameter-efficient fine-tuning approaches.
  • All experiments are conducted using a single pathology foundation model, GigaPath, while the paper makes broader claims about pathology foundation models and pretrained vision transformers.
  • It is unclear why the proposed adaptation strategies are particularly suitable for pathology foundation models. From the technical description, the methods appear largely applicable to general vision transformers. The paper does not sufficiently explain which challenges are unique to pathology foundation models or how the proposed strategies specifically address these challenges.
  • The experiments are performed on a single NVIDIA H200 GPU. This is a high-end and expensive GPU with substantially more memory than commonly available GPUs. Therefore, the results demonstrate that the framework works on one high-memory GPU, but they do not establish whether it makes foundation model adaptation accessible on more commonly available single-GPU hardware.
  • The experimental comparisons are potentially confounded by differences in the number of tiles used across configurations. When different methods are evaluated using different tile budgets, it becomes difficult to determine whether performance differences arise from the adaptation strategy or from the additional image information available to the model.
  • Ideally, the authors should report results across multiple random seeds, including mean performance, standard deviations, and confidence intervals where appropriate.
  • Although the memory decomposition is conceptually useful, some of the notation and assumptions are not sufficiently clear.
  • The ability to process more tiles per WSI does not necessarily imply a reduction in overall training cost. Increasing the tile budget may increase computation, training time, and GPU utilization even when peak memory consumption is reduced.
  • The paper states that the hybrid configuration enables "full-dataset training with 1,000 tiles per WSI." However, the training procedure still samples 1,000 tiles from each WSI rather than necessarily using all available tissue tiles. Therefore, this statement may be misleading if readers interpret it as processing every tissue tile in each whole-slide image.
  • The ability to process 1,000 tiles per WSI does not automatically demonstrate that the additional tiles provide more clinically relevant information. Although a larger tile budget may improve tissue coverage, the paper does not clearly establish whether this translates into consistent improvements in predictive performance or clinical utility.
Questions for authors
  1. What is the main technical novelty of TAPFM beyond combining existing techniques such as LoRA and layer freezing?
  2. What challenges specific to pathology foundation models does the proposed method address?
  3. Can the authors provide comparisons using the same tile budget to ensure a fair evaluation of different adaptation strategies?
  4. Can the method work on more commonly available GPUs with 24 GB or 48 GB of memory, rather than an H200?
  5. Does processing more tiles consistently improve prediction accuracy, and how does it affect overall training time and computational cost?
Reviewer_jhRhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends TAPFM to increase the feasible training tile budget for single-GPU adaptation of pathology foundation models. It compares all-block LoRA, dense fine-tuning of only the final N transformer blocks, and LoRA restricted to those final blocks. A CKA-based representation-drift diagnostic uses a short full-fine-tuning probe to select N. Experiments use GigaPath for FGFR3 mutation prediction in bladder cancer and EGFR mutation prediction in lung adenocarcinoma. The authors demonstrate training with 1000 tiles per WSI on one 141GB H200 and report trade-offs between memory use and predictive performance.

Critical strengths
  1. The paper addresses a practical bottleneck in whole-slide task adaptation. It distinguishes trainable-parameter and optimizer memory from the activations retained for encoder backpropagation, explaining how freezing the entire prefix and restricting LoRA to the final blocks reduce complementary memory costs.

  2. The evaluation includes two mutation-prediction tasks and external cohorts from TCGA and Institute 2. Matched-budget measurements provide concrete evidence of memory savings: at 500 tiles, adding LoRA to Freeze-N reduces peak allocated memory from 73.47 to 57.20 GiB. The reported FGFR3 results are promising.

Critical weaknesses
  1. The methodological contribution appears incremental, combining established layer-selective freezing, LoRA, and CKA within TAPFM. Table 2 compares the proposed selection rule with brute-force choices on two tasks, but its added value over simpler layer-selection choices and its generalizability remain insufficiently established. Evaluation is restricted to one foundation model and two mutation-prediction tasks.

  2. The performance and capacity comparisons are insufficiently controlled. Baseline provenance and evaluation populations require clarification, while Table 4 changes adaptation strategy and tile budget together. The submission reports one seed and time-limited runs, without uncertainty estimates for the new AUC results. The claimed increase from 100 to 1000 tiles also relies on TAPFM's original 80GB H100 budget while using a 141GB H200, even though Section 6.2 shows that full fine-tuning with 200 tiles fits on the H200. These issues limit attribution of the gains and confidence in the claimed capacity improvement.

Questions for authors
  1. Baseline provenance and evaluation populations need clarification. Section 5 says TAPFM is reproduced, whereas Table 4 attributes the first two rows to the prior paper. Their eight shared Institute 1/TCGA values exactly match Table 1 of [1], which reports five-fold means with 95% confidence intervals. Please distinguish quoted results, newly trained reproductions, and evaluations of existing checkpoints, including the provenance of the Institute 2 entries. Reconcile TCGA-LUAD counts of 438 versus 463 and original institutional test cohorts of 194/876 versus the current validation cohorts of 194/873. Ensure that all methods are evaluated on the same samples and distinguish checkpoint-selection validation results from held-out test performance.

  2. Table 4 uses 100, 300, 500, and 1000 training tiles for different adaptation strategies, preventing clean attribution of the performance differences. Please provide controlled comparisons separating adaptation strategy, tile budget, and training budget. Report completed epochs or updates, elapsed training time, and stopping criteria for each configuration. Appendix A.3 explicitly states that several runs were limited by wall-clock availability and that convergence is not claimed. Please also provide uncertainty estimates or appropriate statistical analysis for the AUC differences, including the large FGFR3 gains and smaller or negative EGFR changes.

  3. Original TAPFM used an 80GB H100. This submission uses a 141GB H200 and reports full fine-tuning with 200 tiles at 112.88 GiB of peak allocated memory. The 100-tile budget is therefore not its H200 maximum. Please measure each method's maximum feasible tile budget on the same hardware and evaluate under an 80GB memory budget. Please also compare the proposed methods with a TAPFM baseline that uses activation checkpointing. Run this comparison on the same GPU and report AUC, peak GPU memory, maximum training tiles per WSI, and measured training time. If checkpointing cannot be implemented in TAPFM, explain the specific technical obstacle.

  4. CKA is an established measure of representation similarity. Please explain what is new in the cumulative-drift selection rule and show what it adds over simpler choices of trainable layers. Please evaluate whole-slide image classification tasks beyond mutation prediction, such as histological subtyping or prognosis, to test whether the adaptation benefits and selection rule generalize.

  5. Please report the number of probe updates, the sizes and construction of the calibration and held-out probe sets, and their relationship to training/validation/test partitions. Measure the calibration cost and include it in the efficiency assessment. Reconcile N = 11/9 in Section 6.1 with N = 10 in Table 3 and the hybrid description in Appendix A.3, and identify N for each performance and memory result. Please also introduce "Centered Kernel Alignment (CKA)" at its first mention, on page 3, line 127.

[1] Kumar, Neeraj, and Chad Vanderbilt. "Single GPU Task Adaptation of Pathology Foundation Models for Whole Slide Image Analysis." Advances in Neural Information Processing Systems 38 (2026): 11323-11350.

Reviewer_BBfhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigate memory optimization for GigaPath, a digital pathology model for whole-slide images (WSIs). They explore three memory-saving strategies for TAPFM. These involve incorporating low-rank adaptation (LoRA) into the TAPFM update procedure, freezing all Transformer blocks except the last N blocks, and combining these two approaches. The proposed methods, LoRA-TAPFM, Freeze-N-TAPFM, and Freeze-N+LoRA-TAPFM, are evaluated on the BLCA-FGFR3 and LUAD-EGFR mutation prediction tasks. The experimental results demonstrate that these strategies reduce memory usage while maintaining mutation prediction performance.

Critical strengths

I consider this research direction promising, as whole-slide digital pathology models are an active area of research, and reducing their memory requirements remains an important challenge. Overall, the experimental design is comprehensive, and the results are encouraging. For example, Table 3 shows that, with N = 10, the number of trainable parameters is substantially reduced compared with TAPFM.

Critical weaknesses
  1. Although LoRA-TAPFM, Freeze-N-TAPFM, and Freeze-N+LoRA-TAPFM are well motivated, their core methodological ideas appear to rely largely on combining and adapting existing techniques. Both LoRA and Freeze-N, for example, are established approaches to model fine-tuning and memory reduction. This does not mean that combining existing techniques cannot constitute a strong contribution, particularly given the impressive experimental results presented in this study.

  2. The writing would benefit from further refinement. Some details require clearer explanations. In particular, the descriptions of N, the number of trainable layers, appear to be inconsistent. I recommend that the authors check the main text against the appendix and explicitly specify the configuration used in each experiment.

Questions for authors

1.Please articulate the key novel contributions more precisely and explain the motivation behind the proposed methods in greater depth.

  1. What values of N were used for the Freeze-N and hybrid methods in Table 4? How do the settings N* = 11 and N* = 9 in the main text relate to N = 10 in the appendix?

  2. Could the authors include one or two state-of-the-art memory-saving methods for TAPFM, if available, as additional baselines to provide stronger evidence of the effectiveness of the proposed approaches?

  3. Although the manuscript has been polished using generative AI tools, I recommend further revisions to improve its clarity and readability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.